Re: [Cerowrt-devel] Fwd: Dave's wishlist [was: Source-specific routing merged]

2015-03-17 Thread David Lang

On Tue, 17 Mar 2015, Dave Taht wrote:


My quest is always for an extra 9 of reliability. Anyplace where you can
make something more robust (even if it is out at the .99) level, I
tend to like to do in order to have the highest MTBF possible in
combination with all the other moving parts on the spacecraft (spaceship
earth).


There are different ways to add reliability

one is to try and make sure nothing ever fails

the second is to have a way of recovering when things go wrong.


Bufferbloat came about because people got trapped into the first mode of 
thinking (packets should never get lost), when the right answer ended up being 
to realize that we have a recovery method and use it.


Sometimes trying to make sure nothing ever fails adds a lot of complexity to the 
code to handle all the corner cases, and the overall reliability will improve by 
instead simplify normal flow, even if it add a small number of failures, if that 
means that you can have a common set of recovery code that's well excercised and 
tested.


As you are talking about loosing packets with route changes, watch out that you 
don't fall into this trap.


David Lang
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel


Re: [Cerowrt-devel] Fwd: Dave's wishlist [was: Source-specific routing merged]

2015-03-17 Thread dpreed
I agree wholeheartedly with your point, David.

One other clarifying point (I'm not trying to be pedantic, here, but it may 
sound that way):

Reliability is not the same as Availability.  The two are quite different.

 Bufferbloat is pretty much an availability issue, not a reliability issue.  
In other words, packets are not getting lost.  The system is just preventing 
desired use.

Availability issues can be due to actual failures of components, but there are 
lots of availability issues that are caused (as you suggest) by attempts to 
focus narrowly on loss of data or component failures.

When you build a system, there is a temptation to apply what is called the 
Fallacy of Composition (look it up on Wikipedia for precise definition).  The 
key thing in the Fallacy of Composition is that when a system of components has 
a property as a whole, then every component of the system must by definition 
have that property.

(The end-to-end argument is a specific rule that is based on a recognition of 
the Fallacy of Composition in one case.)

We all know that there is never a single moment when any moderately large part 
of the Internet does not contain failed components.  Yet the Internet has 
*very* high availability - 24x7x365, and we don't need to know very much about 
what parts are failing.  That's by design, of course. And it is a design that 
does not derive its properties from a trivial notion of proof of correctness, 
or even bug freeness

The relevance of a failure or even a design flaw to system availability is 
a matter of a much bigger perspective of what the system does, and what its 
users perceive as to whether they can get work done.




On Tuesday, March 17, 2015 3:30pm, David Lang da...@lang.hm said:

 On Tue, 17 Mar 2015, Dave Taht wrote:
 
 My quest is always for an extra 9 of reliability. Anyplace where you can
 make something more robust (even if it is out at the .99) level, I
 tend to like to do in order to have the highest MTBF possible in
 combination with all the other moving parts on the spacecraft (spaceship
 earth).
 
 There are different ways to add reliability
 
 one is to try and make sure nothing ever fails
 
 the second is to have a way of recovering when things go wrong.
 
 
 Bufferbloat came about because people got trapped into the first mode of
 thinking (packets should never get lost), when the right answer ended up being
 to realize that we have a recovery method and use it.
 
 Sometimes trying to make sure nothing ever fails adds a lot of complexity to 
 the
 code to handle all the corner cases, and the overall reliability will improve 
 by
 instead simplify normal flow, even if it add a small number of failures, if 
 that
 means that you can have a common set of recovery code that's well excercised 
 and
 tested.
 
 As you are talking about loosing packets with route changes, watch out that 
 you
 don't fall into this trap.
 
 David Lang
 ___
 Cerowrt-devel mailing list
 Cerowrt-devel@lists.bufferbloat.net
 https://lists.bufferbloat.net/listinfo/cerowrt-devel
 


___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel


[Cerowrt-devel] Fwd: Dave's wishlist [was: Source-specific routing merged]

2015-03-17 Thread Dave Taht
-- Forwarded message --
From: Dave Taht dave.t...@gmail.com
Date: Tue, Mar 17, 2015 at 7:41 AM
Subject: Re: Dave's wishlist [was: Source-specific routing merged]
To: Juliusz Chroboczek j...@pps.univ-paris-diderot.fr
Cc: babel-us...@lists.alioth.debian.org 
babel-us...@lists.alioth.debian.org, Gabriel Kerneis gabr...@kerneis.info,
Steven Barth cy...@openwrt.org, Henning Rogge hro...@gmail.com, Paul
McKenney paul...@linux.vnet.ibm.com, Felix Fietkau n...@nbd.name


My quest is always for an extra 9 of reliability. Anyplace where you can
make something more robust (even if it is out at the .99) level, I
tend to like to do in order to have the highest MTBF possible in
combination with all the other moving parts on the spacecraft (spaceship
earth).

One of the reasons why I love paul mckenney so much is that he deeply cares
about stuff that happens only one in a billion times.

From this blog post of his: http://paulmck.livejournal.com/37782.html

I quickly learned that the bug is difficult to reproduce, requiring
something like 100 hours of focused rcutorture testing. Bisection based on
100-hour tests would have consumed the remainder of 2014 and a significant
fraction of 2015, so something better was required. In fact, something *way*
better was required because there was only a very small number of failures,
which meant that the expected test time to reproduce the bug might well
have been 200 hours or even 300 hours instead of my best guess of 100
hours.

so, thus, I get picky on system daemons.

On Tue, Mar 17, 2015 at 4:45 AM, Juliusz Chroboczek 
j...@pps.univ-paris-diderot.fr wrote:

  1) Did the issue with procd ever get resolved? (sighup I think it was)

 Gabriel, Steven?  Can procd be configured not to send SIGHUP, or shall
 I add an option to babeld to ignore it?  (Currently babeld terminates on
 SIGHUP, and I like it that way, since it prevents a babeld from sticking
 around after you log off.)


this basically logjammed on this issue. Either procd needed to be modified
to be able to
send an arbitrary signal, or babel changed to take sighup as a reload.

I'd done the simple patch to babel.

But I understand your use case also (stop routing on hup via remote
access), and have never poked into procd. Perhaps it could be changed to
take a var for the actual signal number it uses per daemon, but I will
argue that has overhead the openwrt devs would be loathe to take. But: I
will look. Babel can't be the only thing that needs a different signal to
reload...

a third way out is just to patch babel for openwrt...


  2) got the new vars into openwrt, or shall I do?


rtt branch had 2 new vars as best as I recall, and I envisioned the babels
package being retired, which as best as I recall has extra vars like src-eq.

Secondly the command lines would get complex on me, and I figured just
re-writing the conf file was saner than command line args. Get a
sigWHATEVER reload (or mmap) the conf file, checksum it against the
previous version, do nothing if it didn't change.

Thirdly, having an openwrt specific uci and/or ubus parser that could be
compiled in would be more reliable than a script, simpler and faster. I can
try to find funding for doing that... (in like 1.7's timeframe!!) I looked
over the libubus and libuci interfaces and staggered away confused.

Gabriel?

  ecn

 No, since I don't understand why you think that setting ECN on Babel
 packets makes sense.  (It might make sense to set ECN on some Babel
 packets -- the ones that are marked as urgent -- but I'm interested in
 hearing your reasoning.)


fq_codel is the default on openwrt. ECN is enabled on that. Basic ECN
marking is 2 characters of new code. (tracking it harder, but that's
boilerplate code now). hnetd is presently very dumb about coalescing /60s
out of /64s...  I'd like to be trying much faster update schedules on
ethernet as per some of the discussion on homenet.

But, let me take this subject to another thread than this.



  atomic route updates

 Ausgeschlossen.


Nothing is impossible.


 Last time I tried, I got a number of complaints that it
 broke operational networks.


As the new FIB table patches have landed in linux 4.0 and later, it has
done some odd things with RCU that I am not sure would be a good thing with
the present delete+add routes system everything like quagga+ babel seems to
use.

I'd written about it here while discussing the amazing new FIB patches (7x
reductions in lookup time or more), but was not aware that henning had
actually got atomic route updates that worked.

http://lists.openwall.net/netdev/2015/03/11/136

So, perhaps autodetection of some sort here, also, would be of help.

And figuring out why it used to break.

And ooh! atomic route changes! no packet loss at all! Look at that extra 9!


 It's also less important than it used to be, since the hysteresis
 algorithm in 1.5.0 dramatically reduced the number of route switches --
 current versions of Babel should not be loosing a