Re: [rfc] migrate lagg to an rmlock

[Scott Long]

On Aug 29, 2013, at 7:42 AM, John Baldwin  wrote:

> On Saturday, August 24, 2013 10:16:33 am Robert Watson wrote:
>> There are a number of other places in the kernel where migration to an 
>> rmlock 
>> makes sense -- however, some care must be taken for four reasons: (1) while 
>> read locks don't experience line contention, write locking becomes 
>> observably 
>> e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks, unlike 
>> rwlocks, 
>> more expensive so is not suitable for all rwlock line contention spots -- 
>> implement reader priority propagation, so you must reason about; and (3) 
>> historically, rmlocks have not fully implemented WITNESS so you may get less 
>> good debugging output.  if_lagg is a nice place to use rmlocks, as 
>> reconfigurations are very rare, and it's really all about long-term data 
>> stability.
> 
> 3) should no longer be an issue.  rmlocks now have full WITNESS and assertion
> support (including an rm_assert).
> 
> However, one thing to consider is that rmlocks pin readers to CPUs while the
> read lock is held (which rwlocks do not do).

And this is not a problem for the application that we're giving it in the lagg 
driver.

Scott

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [rfc] migrate lagg to an rmlock

[John Baldwin]

On Thursday, August 29, 2013 11:37:08 am Scott Long wrote:
> 
> On Aug 29, 2013, at 7:42 AM, John Baldwin  wrote:
> 
> > On Saturday, August 24, 2013 10:16:33 am Robert Watson wrote:
> >> There are a number of other places in the kernel where migration to an 
> >> rmlock 
> >> makes sense -- however, some care must be taken for four reasons: (1) 
> >> while 
> >> read locks don't experience line contention, write locking becomes 
> >> observably 
> >> e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks, unlike 
> >> rwlocks, 
> >> more expensive so is not suitable for all rwlock line contention spots -- 
> >> implement reader priority propagation, so you must reason about; and (3) 
> >> historically, rmlocks have not fully implemented WITNESS so you may get 
> >> less 
> >> good debugging output.  if_lagg is a nice place to use rmlocks, as 
> >> reconfigurations are very rare, and it's really all about long-term data 
> >> stability.
> > 
> > 3) should no longer be an issue.  rmlocks now have full WITNESS and 
> > assertion
> > support (including an rm_assert).
> > 
> > However, one thing to consider is that rmlocks pin readers to CPUs while the
> > read lock is held (which rwlocks do not do).
> 
> And this is not a problem for the application that we're giving it in the
> lagg driver.

That is likely true.  I was merely tweaking Robert's general guidelines re: 
rmlock.

-- 
John Baldwin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [rfc] migrate lagg to an rmlock

[John Baldwin]

On Saturday, August 24, 2013 10:16:33 am Robert Watson wrote:
> There are a number of other places in the kernel where migration to an rmlock 
> makes sense -- however, some care must be taken for four reasons: (1) while 
> read locks don't experience line contention, write locking becomes observably 
> e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks, unlike rwlocks, 
> more expensive so is not suitable for all rwlock line contention spots -- 
> implement reader priority propagation, so you must reason about; and (3) 
> historically, rmlocks have not fully implemented WITNESS so you may get less 
> good debugging output.  if_lagg is a nice place to use rmlocks, as 
> reconfigurations are very rare, and it's really all about long-term data 
> stability.

3) should no longer be an issue.  rmlocks now have full WITNESS and assertion
support (including an rm_assert).

However, one thing to consider is that rmlocks pin readers to CPUs while the
read lock is held (which rwlocks do not do).

-- 
John Baldwin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [rfc] migrate lagg to an rmlock

[Davide Italiano]

On Sat, Aug 24, 2013 at 7:16 AM, Robert Watson  wrote:
> On Sat, 24 Aug 2013, Alexander V. Chernikov wrote:
>
>> On 24.08.2013 00:54, Adrian Chadd wrote:
>>>
>>>
>>> I'd like to commit this to -10. It migrates the if_lagg locking
>>> from a rw lock to a rm lock. We see a bit of contention between the
>>> transmit and
>>
>>
>> We're running lagg with rmlock on several hundred heavily loaded machines,
>> it really works better. However, there should not be any contention between
>> receive and transmit side since there is actually no _real_ need to lock RX
>> (and even use lagg receive code at all):
>>
>> http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html
>
>
> We should distinguish "lock contention" from "line contention".  When
> acquiring a rwlock on multiple CPUs concurrently, the cache lines used to
> implement the lock are contended, as they must bounce between caches via the
> cache coherence protocol, also referred to as "contention".  In the if_lagg
> code, I assume that the read-only acquire of the rwlock (and perhaps now
> rmlock) is for data stability rather than mutual exclusion -- e.g., to allow
> processing to completion against a stable version of the lagg configuration.
> As such, indeed, there should be no lock contention unless a configuration
> update takes place, and any line contention is a property of the locking
> primitive rather than data model.
>
> There are a number of other places in the kernel where migration to an
> rmlock makes sense -- however, some care must be taken for four reasons: (1)
> while read locks don't experience line contention, write locking becomes
> observably e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks,
> unlike rwlocks, more expensive so is not suitable for all rwlock line
> contention spots -- implement reader priority propagation, so you must
> reason about; and (3) historically, rmlocks have not fully implemented
> WITNESS so you may get less good debugging output.  if_lagg is a nice place

I'm not sure what you mean here with (3), because from my
understanding of the code WITNESS is implemented both in the sleepable
and non-sleepable case, but there could be something I'm missing.
Something I think we lack in rmlock code is fully supported
LOCK_PROFILING as we have in all the other primitives, but again, if
I'm wrong feel free to correct me.

> to use rmlocks, as reconfigurations are very rare, and it's really all about
> long-term data stability.
>
> Robert
>
> ___
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Thanks,

-- 
Davide

"There are no solved problems; there are only problems that are more
or less solved" -- Henri Poincare
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [rfc] migrate lagg to an rmlock

[Scott Long]

On Aug 24, 2013, at 12:14 PM, Alfred Perlstein  wrote:

> On 8/24/13 10:47 AM, Robert N. M. Watson wrote:
>> On 24 Aug 2013, at 17:36, Alfred Perlstein wrote:
>> 
 We should distinguish "lock contention" from "line contention". When 
 acquiring a rwlock on multiple CPUs concurrently, the cache lines used to 
 implement the lock are contended, as they must bounce between caches via 
 the cache coherence protocol, also referred to as "contention".  In the 
 if_lagg code, I assume that the read-only acquire of the rwlock (and 
 perhaps now rmlock) is for data stability rather than mutual exclusion -- 
 e.g., to allow processing to completion against a stable version of the 
 lagg configuration. As such, indeed, there should be no lock contention 
 unless a configuration update takes place, and any line contention is a 
 property of the locking primitive rather than data model.
 
 There are a number of other places in the kernel where migration to an 
 rmlock makes sense -- however, some care must be taken for four reasons: 
 (1) while read locks don't experience line contention, write locking 
 becomes observably e.g., rmlocks might not be suitable for tcbinfo; (2) 
 rmlocks, unlike rwlocks, more expensive so is not suitable for all rwlock 
 line contention spots -- implement reader priority propagation, so you 
 must reason about; and (3) historically, rmlocks have not fully 
 implemented WITNESS so you may get less good debugging output.  if_lagg is 
 a nice place to use rmlocks, as reconfigurations are very rare, and it's 
 really all about long-term data stability.
>>> Robert, what do you think about a quick swap of the ifnet structures to 
>>> counter before 10.x?
>> Could you be more specific about the proposal you're making?
>> 
>> Robert
> 
> The lagg patch referred to in the thread seems to indicate that zero locking 
> is needed if we just switched to counter(9), that makes me wonder if we could 
> do better with locking in other places if we switched to counter(9) while we 
> have the chance.
> 
> This is the thread:
> 
> http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html
> 
>> /
> />/Perfect solution would be to convert ifnet(9) to counters(9), but this
> />/requires much more work, and unfortunately ABI change, so temporarily
> />/patch lagg(4) manually.
> />//>/We store counters in the softc, and once per second push their 
> values
> />/to legacy ifnet counters./
> 

Some sort of gatekeeper semantic is needed to ensure that configuration changes 
to the lagg state don't cause incorrect behavior to the data path.  It's not 
about protecting the integrity of counters.  This can be done several ways, but 
right now it's via a read/write semantic.

Scott

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [rfc] migrate lagg to an rmlock

[Adrian Chadd]

Hihi,

There's two parts to my patch:

* one is migrating the rwlock to rmlock - not because of counters, but
because the lock is protecting consistency when changing the lagg config
* one is adding a new lock specifically to ensure that the callout is
atomically created/called/destroyed

The latter has nothing to do with the actual counters - they're already
using the atomic counter API, so the lock doesn't need to be held just to
read them for _counter_ consistency. It's just held to make sure the
callout is never called parallel to the destruction of the lagg interface.



-adrian
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [rfc] migrate lagg to an rmlock

[Alfred Perlstein]

On 8/24/13 10:47 AM, Robert N. M. Watson wrote:

On 24 Aug 2013, at 17:36, Alfred Perlstein wrote:


We should distinguish "lock contention" from "line contention". When acquiring a rwlock 
on multiple CPUs concurrently, the cache lines used to implement the lock are contended, as they must bounce 
between caches via the cache coherence protocol, also referred to as "contention".  In the if_lagg 
code, I assume that the read-only acquire of the rwlock (and perhaps now rmlock) is for data stability rather 
than mutual exclusion -- e.g., to allow processing to completion against a stable version of the lagg 
configuration. As such, indeed, there should be no lock contention unless a configuration update takes place, 
and any line contention is a property of the locking primitive rather than data model.

There are a number of other places in the kernel where migration to an rmlock 
makes sense -- however, some care must be taken for four reasons: (1) while 
read locks don't experience line contention, write locking becomes observably 
e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks, unlike rwlocks, 
more expensive so is not suitable for all rwlock line contention spots -- 
implement reader priority propagation, so you must reason about; and (3) 
historically, rmlocks have not fully implemented WITNESS so you may get less 
good debugging output.  if_lagg is a nice place to use rmlocks, as 
reconfigurations are very rare, and it's really all about long-term data 
stability.

Robert, what do you think about a quick swap of the ifnet structures to counter 
before 10.x?

Could you be more specific about the proposal you're making?

Robert


The lagg patch referred to in the thread seems to indicate that zero 
locking is needed if we just switched to counter(9), that makes me 
wonder if we could do better with locking in other places if we switched 
to counter(9) while we have the chance.


This is the thread:

http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html

/

/>/Perfect solution would be to convert ifnet(9) to counters(9), but this
/>/requires much more work, and unfortunately ABI change, so temporarily
/>/patch lagg(4) manually.
/>/
/>/We store counters in the softc, and once per second push their values

/>/to legacy ifnet counters./



--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [rfc] migrate lagg to an rmlock

[Robert N. M. Watson]

On 24 Aug 2013, at 17:36, Alfred Perlstein wrote:

>> We should distinguish "lock contention" from "line contention". When 
>> acquiring a rwlock on multiple CPUs concurrently, the cache lines used to 
>> implement the lock are contended, as they must bounce between caches via the 
>> cache coherence protocol, also referred to as "contention".  In the if_lagg 
>> code, I assume that the read-only acquire of the rwlock (and perhaps now 
>> rmlock) is for data stability rather than mutual exclusion -- e.g., to allow 
>> processing to completion against a stable version of the lagg configuration. 
>> As such, indeed, there should be no lock contention unless a configuration 
>> update takes place, and any line contention is a property of the locking 
>> primitive rather than data model.
>> 
>> There are a number of other places in the kernel where migration to an 
>> rmlock makes sense -- however, some care must be taken for four reasons: (1) 
>> while read locks don't experience line contention, write locking becomes 
>> observably e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks, 
>> unlike rwlocks, more expensive so is not suitable for all rwlock line 
>> contention spots -- implement reader priority propagation, so you must 
>> reason about; and (3) historically, rmlocks have not fully implemented 
>> WITNESS so you may get less good debugging output.  if_lagg is a nice place 
>> to use rmlocks, as reconfigurations are very rare, and it's really all about 
>> long-term data stability.
> 
> Robert, what do you think about a quick swap of the ifnet structures to 
> counter before 10.x?

Could you be more specific about the proposal you're making?

Robert
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [rfc] migrate lagg to an rmlock

[Alfred Perlstein]

On 8/24/13 7:16 AM, Robert Watson wrote:

On Sat, 24 Aug 2013, Alexander V. Chernikov wrote:


On 24.08.2013 00:54, Adrian Chadd wrote:


I'd like to commit this to -10. It migrates the if_lagg locking
from a rw lock to a rm lock. We see a bit of contention between the
transmit and


We're running lagg with rmlock on several hundred heavily loaded 
machines, it really works better. However, there should not be any 
contention between receive and transmit side since there is actually 
no _real_ need to lock RX (and even use lagg receive code at all):


http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html


We should distinguish "lock contention" from "line contention". When 
acquiring a rwlock on multiple CPUs concurrently, the cache lines used 
to implement the lock are contended, as they must bounce between 
caches via the cache coherence protocol, also referred to as 
"contention".  In the if_lagg code, I assume that the read-only 
acquire of the rwlock (and perhaps now rmlock) is for data stability 
rather than mutual exclusion -- e.g., to allow processing to 
completion against a stable version of the lagg configuration. As 
such, indeed, there should be no lock contention unless a 
configuration update takes place, and any line contention is a 
property of the locking primitive rather than data model.


There are a number of other places in the kernel where migration to an 
rmlock makes sense -- however, some care must be taken for four 
reasons: (1) while read locks don't experience line contention, write 
locking becomes observably e.g., rmlocks might not be suitable for 
tcbinfo; (2) rmlocks, unlike rwlocks, more expensive so is not 
suitable for all rwlock line contention spots -- implement reader 
priority propagation, so you must reason about; and (3) historically, 
rmlocks have not fully implemented WITNESS so you may get less good 
debugging output.  if_lagg is a nice place to use rmlocks, as 
reconfigurations are very rare, and it's really all about long-term 
data stability.


Robert
___
freebsd-...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"



Robert, what do you think about a quick swap of the ifnet structures to 
counter before 10.x?


-Alfred

--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [rfc] migrate lagg to an rmlock

[Adrian Chadd]

Sorry, I meant "line contention" rather than "lock contention". Yes, you're
right.



-adrian



On 24 August 2013 07:16, Robert Watson  wrote:

> On Sat, 24 Aug 2013, Alexander V. Chernikov wrote:
>
>  On 24.08.2013 00:54, Adrian Chadd wrote:
>>
>>>
>>> I'd like to commit this to -10. It migrates the if_lagg locking
>>> from a rw lock to a rm lock. We see a bit of contention between the
>>> transmit and
>>>
>>
>> We're running lagg with rmlock on several hundred heavily loaded
>> machines, it really works better. However, there should not be any
>> contention between receive and transmit side since there is actually no
>> _real_ need to lock RX (and even use lagg receive code at all):
>>
>> http://lists.freebsd.org/**pipermail/svn-src-all/2013-**April/067570.html
>>
>
> We should distinguish "lock contention" from "line contention".  When
> acquiring a rwlock on multiple CPUs concurrently, the cache lines used to
> implement the lock are contended, as they must bounce between caches via
> the cache coherence protocol, also referred to as "contention".  In the
> if_lagg code, I assume that the read-only acquire of the rwlock (and
> perhaps now rmlock) is for data stability rather than mutual exclusion --
> e.g., to allow processing to completion against a stable version of the
> lagg configuration. As such, indeed, there should be no lock contention
> unless a configuration update takes place, and any line contention is a
> property of the locking primitive rather than data model.
>
> There are a number of other places in the kernel where migration to an
> rmlock makes sense -- however, some care must be taken for four reasons:
> (1) while read locks don't experience line contention, write locking
> becomes observably e.g., rmlocks might not be suitable for tcbinfo; (2)
> rmlocks, unlike rwlocks, more expensive so is not suitable for all rwlock
> line contention spots -- implement reader priority propagation, so you must
> reason about; and (3) historically, rmlocks have not fully implemented
> WITNESS so you may get less good debugging output.  if_lagg is a nice place
> to use rmlocks, as reconfigurations are very rare, and it's really all
> about long-term data stability.
>
> Robert
>
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [rfc] migrate lagg to an rmlock

[Robert Watson]

On Sat, 24 Aug 2013, Alexander V. Chernikov wrote:


On 24.08.2013 00:54, Adrian Chadd wrote:


I'd like to commit this to -10. It migrates the if_lagg locking
from a rw lock to a rm lock. We see a bit of contention between the
transmit and


We're running lagg with rmlock on several hundred heavily loaded machines, 
it really works better. However, there should not be any contention between 
receive and transmit side since there is actually no _real_ need to lock RX 
(and even use lagg receive code at all):


http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html


We should distinguish "lock contention" from "line contention".  When 
acquiring a rwlock on multiple CPUs concurrently, the cache lines used to 
implement the lock are contended, as they must bounce between caches via the 
cache coherence protocol, also referred to as "contention".  In the if_lagg 
code, I assume that the read-only acquire of the rwlock (and perhaps now 
rmlock) is for data stability rather than mutual exclusion -- e.g., to allow 
processing to completion against a stable version of the lagg configuration. 
As such, indeed, there should be no lock contention unless a configuration 
update takes place, and any line contention is a property of the locking 
primitive rather than data model.


There are a number of other places in the kernel where migration to an rmlock 
makes sense -- however, some care must be taken for four reasons: (1) while 
read locks don't experience line contention, write locking becomes observably 
e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks, unlike rwlocks, 
more expensive so is not suitable for all rwlock line contention spots -- 
implement reader priority propagation, so you must reason about; and (3) 
historically, rmlocks have not fully implemented WITNESS so you may get less 
good debugging output.  if_lagg is a nice place to use rmlocks, as 
reconfigurations are very rare, and it's really all about long-term data 
stability.


Robert
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [rfc] migrate lagg to an rmlock

[Alexander V. Chernikov]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 24.08.2013 00:54, Adrian Chadd wrote:
> Hi,
> 
> I'd like to commit this to -10. It migrates the if_lagg locking
> from a rw lock to a rm lock. We see a bit of contention between the
> transmit and
We're running lagg with rmlock on several hundred heavily loaded
machines, it really works better.
However, there should not be any contention between receive and
transmit side since there is actually no _real_ need to lock RX (and
even use lagg receive code at all):

http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html

> receive sides of lagg during traffic loads (10+ gigabit per
> second.) Using rmlocks eliminate this.
> 
> http://people.freebsd.org/~adrian/netflix/20130819-lagg-rmlock-1.diff
>
>  Thanks,
> 
> 
> -adrian ___ 
> freebsd-...@freebsd.org mailing list 
> http://lists.freebsd.org/mailman/listinfo/freebsd-net To
> unsubscribe, send any mail to
> "freebsd-net-unsubscr...@freebsd.org"
> 

-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.20 (FreeBSD)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlIYqjYACgkQwcJ4iSZ1q2n9fgCePHOfC3tzBIG54ayNg7d8TKMC
gIMAn2/paUBpDIRVd+3s7snNFCmZNWgd
=i6Ye
-END PGP SIGNATURE-
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

[rfc] migrate lagg to an rmlock

[Adrian Chadd]

Hi,

I'd like to commit this to -10. It migrates the if_lagg locking from a rw
lock to a rm lock. We see a bit of contention between the transmit and
receive sides of lagg during traffic loads (10+ gigabit per second.) Using
rmlocks eliminate this.

http://people.freebsd.org/~adrian/netflix/20130819-lagg-rmlock-1.diff

Thanks,


-adrian
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"