Re: [Lsr] Dynamic flow control for flooding

[tony . li]

Les,

> There is something to be said for simply “flooding fast” and not worrying 
> about flow control at all (regardless of whether TX or RX mechanisms would be 
> used).


The only thing I would say to that is: really bad idea.

If you supersaturate the receiver, you waste transmitter in the transmission, 
you swamp the receiver causing packet loss, you potentially trigger the loss of 
IIH’s, you risk causing a cascade failure, and until we come up with a better 
retransmission mechanism, you probably actually delay network convergence, as 
nothing is going to happen until you’ve completed retransmissions.

The way to maximize goodput is NOT to transmit blindly.


> But most important to me is to recognize that flow control (however done) is 
> not fixing anything – nor making the flooding work better. The network is 
> compromised and flow control won’t fix it.


 The network is not compromised.


> If you accept that, then it makes sense to look for the simplest way to do 
> flow control and that is decidedly not from the RX side. (I expect Tony Li to 
> disagree with that 😊– but I have already outlined why it is more complex to 
> do it from the Rx side.)



Flow control cannot be done without involvement of the RX side.  That’s why 
it’s called flow _control_.  The only thing that can be done purely from the TX 
side is a unilateral (and pessimal) transmit rate cap that will have to allow 
for the worst case CPU in the worst case situation (e.g., BGP impacting the 
CPU).

Flow control is the creation of a control loop and that requires feedback from 
the receiver.  This is true in every form of true flow control: XON/XOFF, 
RTS/CTS, sliding window protocols, credit based fabric mechanisms, etc.  

I’ll go so far as to quote Wikipedia:

"In data communications , 
flow control is the process of managing the rate of data transmission between 
two nodes to prevent a fast sender from overwhelming a slow receiver. It 
provides a mechanism for the receiver to control the transmission speed, so 
that the receiving node is not overwhelmed with data from transmitting node.”

Tony

___
Lsr mailing list
Lsr@ietf.org
https://www.ietf.org/mailman/listinfo/lsr

Re: [Lsr] Dynamic flow control for flooding

[tony . li]

Les,

> I also think we all agree on the goal - which is to flood significantly 
> faster than many implementations do today to handle deployments like the case 
> you mention below.


I agree with that, but you haven’t responded to the goal that I proposed: keep 
the receiver processing PDUs.


> Beyond this point, I have a different perspective.
>  
> As network-wide convergence depends upon fast propagation of LSP changes – 
> which in turn requires consistent flooding rates on all interfaces enabled 
> for flooding – a properly provisioned network MUST be able to sustain a 
> consistent flooding rate or the operation of the network will suffer. We 
> therefore need to view flow control issues as indicative of a problem.
>  
> It is a mistake to equate LSP flooding with a set of independent P2P 
> “connections” – each of which can operate at a rate independent of the other.
>  
> If we can agree on this, then I believe we will have placed the flow control 
> problem in its proper perspective – in which case it will become easier to 
> agree on the best way to implement flow control.


I agree that we want network wide convergence.  However, I disagree that the 
right thing to do is to uniformly flood at the same rate on all interfaces.

First, the rate that you select might be too fast for one neighbor and not for 
the others.  Real flow control would help address this.

Second, all flooding paths are not created equal.  You do not know, a priori, 
how to flood to get uniform network wide propagation.  The variance in CPU 
loading alone is going to cause different routers to flood at different rates, 
and so it may actaully be optimal to flood quickly down one path, knowing that 
the data will reach the other end of the network and flood back before the slow 
CPU can absorb and flood.

Dropping down to the least common denominator CPU speed in the entire network 
is going to be undoable without an oracle, and absurdly slow even with that.

Tony

___
Lsr mailing list
Lsr@ietf.org
https://www.ietf.org/mailman/listinfo/lsr

Re: [Lsr] Dynamic flow control for flooding

[Les Ginsberg (ginsberg)]

Stephane –

There is much we agree on.

There is something to be said for simply “flooding fast” and not worrying about 
flow control at all (regardless of whether TX or RX mechanisms would be used). 
Some packets would be dropped, but retransmission timers will insure that the 
flooding eventually succeeds and retransmit timers are long (5 seconds by 
default). (I am not the only one mentioning this BTW…)

But most important to me is to recognize that flow control (however done) is 
not fixing anything – nor making the flooding work better. The network is 
compromised and flow control won’t fix it.
If you accept that, then it makes sense to look for the simplest way to do flow 
control and that is decidedly not from the RX side. (I expect Tony Li to 
disagree with that 😊 – but I have already outlined why it is more complex to do 
it from the Rx side.)

   Les


From: stephane.litkow...@orange.com 
Sent: Tuesday, July 23, 2019 9:50 PM
To: Les Ginsberg (ginsberg) ; Tony Li ; 
lsr@ietf.org
Subject: RE: [Lsr] Dynamic flow control for flooding

Hi Les,

I agree that flooding is a global thing and not a local mechanism if we 
consider that the ultimate goal is to get the LSDB in-sync as fast as we can on 
all the nodes.

I just want to highlight three things:

  *   Link delay (due to transmission link distance) is already affecting the 
flooding speed (especially when we need to cross some links which have 100msec 
of RTD), so the flooding speed is already not equal on each link
  *   I put this one in parenthesis as it may be controversial ☺ (To converge a 
path after a topology change, we do not always require all the nodes to get the 
LSDB in-sync (I mean from a fwding point of view). That’s a tricky topic 
because it is highly depending on the network topology and in one hand flooding 
one or two hops away allows to converge the path, while in an other hand, it 
may create microloops with another network design. )
  *   I’m really wondering how much difference we may have considering the 
different routers we have in a single area today. Even if we have some legacy 
routers still deployed, they are more powerful compared to the time the ISO 
spec was done. Are we expecting hundreds of msec difference or tens between 
last generation of routers deployed and the legacy one ? In addition, in our 
case, we try to create consistent design, which means that we are trying to 
avoid having legacy routers in transit between last generation of routers and 
we are pushing the legacy one at the edge or try to remove them. There may be 
some transient situation when it happens but that’s not a design goal. This is 
to say that I’m not hurted to get a very fast flooding value on my core and 
last generation edges while letting a more conservative value for legacy edges. 
And I’m not expecting to have so much differences between the two (at least not 
really more than the link delay that may already exists and impact flooding).

Another point is that I would be really glad to see how much the flooding time 
is impacting the convergence time in real networks taking into account that the 
FIB rewrite is usually the biggest contributor (unfortunately we don’t have 
really instrumentation today to measure flooding). I’m not telling that there 
is nothing to do, of course the default flooding time we had for years could be 
improved and I fully agree. I’m just always interested to have some potential 
gain measurement.

Flow control is required in any case, we can always find a case when the IS-IS 
process will not get enough CPU time because CPU is busy doing other stuffs and 
IS-IS can’t process the input PDUs (as an example).


Brgds,

From: Lsr [mailto:lsr-boun...@ietf.org] On Behalf Of Les Ginsberg (ginsberg)
Sent: Tuesday, July 23, 2019 16:30
To: Tony Li; lsr@ietf.org
Subject: Re: [Lsr] Dynamic flow control for flooding

Tony –

Thanx for picking up the discussion.
Thanx also for doing the math to show that bandwidth is not a concern. I think 
most/all of us knew that – but it is good to put that small question behind us.

I also think we all agree on the goal - which is to flood significantly faster 
than many implementations do today to handle deployments like the case you 
mention below.

Beyond this point, I have a different perspective.

As network-wide convergence depends upon fast propagation of LSP changes – 
which in turn requires consistent flooding rates on all interfaces enabled for 
flooding – a properly provisioned network MUST be able to sustain a consistent 
flooding rate or the operation of the network will suffer. We therefore need to 
view flow control issues as indicative of a problem.

It is a mistake to equate LSP flooding with a set of independent P2P 
“connections” – each of which can operate at a rate independent of the other.

If we can agree on this, then I believe we will have placed the flow control 
problem in its proper perspective – in which case it will become easier

Re: [Lsr] Dynamic flow control for flooding

[tony . li]

Hi Robert,


> Are you working on the assumption that there is no data link layer flow 
> control already which could signal the OS congestion on the receiving end ? 


I am assuming that there is no link layer flow control.  I can’t recall working 
on a system that supports X.25 since about 1995, so I don’t think that’s a 
common use case today. 


> Are we also working on the assumptions that when ISIS PDUs are send in DCs 
> (unknown today case when out of the sudden 1000s of LSPs may need to get 
> flooded) use of some L4 fancy flow control is an overkill and we must invent 
> new essentially L2 flow control to cover this case to address partition 
> repair ? 


I am not assuming that the issue is restricted to the DC. Flooding is an issue 
in all IS-IS networks.

1000s of LSPs can occur in any IS-IS network of significant scale.  All it 
takes is healing a partition and there can easily be a large number of LSPs to 
transmit.  The case of 1000s of LSPs is of interest because the scale magnifies 
the flooding problem.  If we only have one LSP that needs flooding, this entire 
discussion is moot.

I am assuming that we want to go faster.  That does seem to be something that 
we have agreement on.

I am assuming that we dont’ want to go too fast.  Overrunning the receiver is 
wasteful. I think we all agree on that.

I am not assuming that we have to use some ‘L4 fancy flow control’, but I am 
trying to get a reasonable approximation to optimal goodput, with errors being 
on the conservative side (i.e., not overrunning the receiver).

My understanding of control theory is pretty rudimentary, but what I do know is 
that it is going to be very difficult to achieve high goodput without a control 
loop of some flavor. I’m open to how we do this.  Henk proposed that we simply 
pick up TCP for this, but my concern with that is really about introducing a 
whole new dependency to the protocol.  That’s a lot to chew.  Do we really need 
it all? I hope not.  Thus, Bruno’s original suggestion sparked my interest in 
doing something dynamic and simple.

Tony


___
Lsr mailing list
Lsr@ietf.org
https://www.ietf.org/mailman/listinfo/lsr

Re: [Lsr] Dynamic flow control for flooding

[stephane.litkowski]

Hi Les,

I agree that flooding is a global thing and not a local mechanism if we 
consider that the ultimate goal is to get the LSDB in-sync as fast as we can on 
all the nodes.

I just want to highlight three things:

-  Link delay (due to transmission link distance) is already affecting 
the flooding speed (especially when we need to cross some links which have 
100msec of RTD), so the flooding speed is already not equal on each link

-  I put this one in parenthesis as it may be controversial ☺ (To 
converge a path after a topology change, we do not always require all the nodes 
to get the LSDB in-sync (I mean from a fwding point of view). That’s a tricky 
topic because it is highly depending on the network topology and in one hand 
flooding one or two hops away allows to converge the path, while in an other 
hand, it may create microloops with another network design. )

-  I’m really wondering how much difference we may have considering the 
different routers we have in a single area today. Even if we have some legacy 
routers still deployed, they are more powerful compared to the time the ISO 
spec was done. Are we expecting hundreds of msec difference or tens between 
last generation of routers deployed and the legacy one ? In addition, in our 
case, we try to create consistent design, which means that we are trying to 
avoid having legacy routers in transit between last generation of routers and 
we are pushing the legacy one at the edge or try to remove them. There may be 
some transient situation when it happens but that’s not a design goal. This is 
to say that I’m not hurted to get a very fast flooding value on my core and 
last generation edges while letting a more conservative value for legacy edges. 
And I’m not expecting to have so much differences between the two (at least not 
really more than the link delay that may already exists and impact flooding).

Another point is that I would be really glad to see how much the flooding time 
is impacting the convergence time in real networks taking into account that the 
FIB rewrite is usually the biggest contributor (unfortunately we don’t have 
really instrumentation today to measure flooding). I’m not telling that there 
is nothing to do, of course the default flooding time we had for years could be 
improved and I fully agree. I’m just always interested to have some potential 
gain measurement.

Flow control is required in any case, we can always find a case when the IS-IS 
process will not get enough CPU time because CPU is busy doing other stuffs and 
IS-IS can’t process the input PDUs (as an example).


Brgds,

From: Lsr [mailto:lsr-boun...@ietf.org] On Behalf Of Les Ginsberg (ginsberg)
Sent: Tuesday, July 23, 2019 16:30
To: Tony Li; lsr@ietf.org
Subject: Re: [Lsr] Dynamic flow control for flooding

Tony –

Thanx for picking up the discussion.
Thanx also for doing the math to show that bandwidth is not a concern. I think 
most/all of us knew that – but it is good to put that small question behind us.

I also think we all agree on the goal - which is to flood significantly faster 
than many implementations do today to handle deployments like the case you 
mention below.

Beyond this point, I have a different perspective.

As network-wide convergence depends upon fast propagation of LSP changes – 
which in turn requires consistent flooding rates on all interfaces enabled for 
flooding – a properly provisioned network MUST be able to sustain a consistent 
flooding rate or the operation of the network will suffer. We therefore need to 
view flow control issues as indicative of a problem.

It is a mistake to equate LSP flooding with a set of independent P2P 
“connections” – each of which can operate at a rate independent of the other.

If we can agree on this, then I believe we will have placed the flow control 
problem in its proper perspective – in which case it will become easier to 
agree on the best way to implement flow control.

   Les



From: Lsr mailto:lsr-boun...@ietf.org>> On Behalf Of Tony 
Li
Sent: Tuesday, July 23, 2019 6:34 AM
To: lsr@ietf.org
Subject: [Lsr] Dynamic flow control for flooding


Hi all,

I’d like to continue the discussion that we left off with last night.

The use case that I posited was a situation where we had 1000 LSPs to flood. 
This is an interesting case that can happen if there was a large network that 
partitioned and has now healed.  All LSPs from the other side of the partition 
are going to need to be updated.

Let’s further suppose that the LSPs have an average size of 1KB.  Thus, the 
entire transfer is around 1MB.

Suppose that we’re doing this on a 400Gb/s link. If we were to transmit the 
whole batch of LSPs at once, it takes a whopping 20us.  Not milliseconds, 
microseconds.  2x10^-5s.  Clearly, we are not going to be rate limited by 
bandwidth.

Note that 20us is an unreasonable lower bound: we cannot reasonably expect a 
node to absorb 1k PDUs back to 

Re: [Lsr] Dynamic flow control for flooding

[Les Ginsberg (ginsberg)]

David -

Let's take Tony's example test case.

A large network is partitioned and heals. As a result I now have 1000 LSPs 
which need to be flooded.

Let's suppose on most links/nodes in the network I can support receiving of 500 
LSPs/second.
But on one link/node I can only support receiving 33 LSPs/second.

This means we are at risk for part of the network converging the LSPDB in 2+ 
seconds and part of the network converging the LSPDB in 33+ seconds.

Having the means for the node which only supports 33 LSPs/second to signal its 
"upstream neighbors" to not overflow its receive queue isn't going to help 
network convergence.

That's all I am saying.

   Les


> -Original Message-
> From: David Lamparter 
> Sent: Tuesday, July 23, 2019 2:14 PM
> To: Les Ginsberg (ginsberg) 
> Cc: Tony Li ; lsr@ietf.org
> Subject: Re: [Lsr] Dynamic flow control for flooding
> 
> Hi Les,
> 
> 
> On Tue, Jul 23, 2019 at 08:29:30PM +, Les Ginsberg (ginsberg) wrote:
> > [...] As network-wide convergence depends upon fast propagation of LSP
> > changes -
> 
> you're losing me between that previous part and the next:
> 
> > - which in turn requires consistent flooding rates on all interfaces
> > enabled for flooding [...]
> 
> I understand and follow your reasoning if we have a classical timer that
> limits flooding rates per LSP.  If we get multiple updates to the same
> LSP, dissimilar flooding rates imply we might just have sent out the
> previous now-outdated state, and we block for some potentially lengthy
> time before sending out the most recent version of that LSP.
> 
> I don't understand how we get delayed propagation of LSP changes if we
> employ some mechanism to raise the flooding rate to something based
> around the target system's capabilities.
> 
> Could you elaborate on how we get delayed LSP propagation in this
> scenario?
> 
> Thanks,
> 
> 
> -David

___
Lsr mailing list
Lsr@ietf.org
https://www.ietf.org/mailman/listinfo/lsr

[Lsr] Fwd: [Rift] RIFT-Python flooding reduction feature guide

[Tony Przygienda]

I read quickly through it, very readable plain narrative explanation of the
IMO best-in-class flood reduction we've in RIFT, implemented and operating
today so I'm fwd'ing through to LSR given the lively discussions around the
topic these days there as well ... Understandably, the spec is dense and
detailed while you have the knack to make it into a digestable English
narrative ;-)

What you omitted is that the solution largely deals with the very difficult
problem of "flooding incast" when a parent node boots up as well by
ignoring first TIRE (request) when not being flood repeater ...

thanks Bruno

--- tony

-- Forwarded message -
From: Bruno Rijsman 
Date: Tue, Jul 23, 2019 at 6:04 PM
Subject: [Rift] RIFT-Python flooding reduction feature guide
To: 


I just posted the feature guide for RIFT-Python flooding reduction:
http://bit.ly/flooding-reduction-feature-guide. This was implemented some
time ago already, but I finally got around to documenting it in preparation
for the RIFT working group meeting on Thursday.

Comments / corrections / pull requests welcome.

— Bruno
___
RIFT mailing list
r...@ietf.org
https://www.ietf.org/mailman/listinfo/rift
___
Lsr mailing list
Lsr@ietf.org
https://www.ietf.org/mailman/listinfo/lsr

Re: [Lsr] Dynamic flow control for flooding

[Robert Raszuk]

Hi Tony,

Are you working on the assumption that there is no data link layer flow
control already which could signal the OS congestion on the receiving end ?

Are we also working on the assumptions that when ISIS PDUs are send in DCs
(unknown today case when out of the sudden 1000s of LSPs may need to get
flooded) use of some L4 fancy flow control is an overkill and we must
invent new essentially L2 flow control to cover this case to address
partition repair ?

Thx,
R.

On Tue, Jul 23, 2019 at 3:34 PM Tony Li  wrote:

>
> Hi all,
>
> I’d like to continue the discussion that we left off with last night.
>
> The use case that I posited was a situation where we had 1000 LSPs to
> flood. This is an interesting case that can happen if there was a large
> network that partitioned and has now healed.  All LSPs from the other side
> of the partition are going to need to be updated.
>
> Let’s further suppose that the LSPs have an average size of 1KB.  Thus,
> the entire transfer is around 1MB.
>
> Suppose that we’re doing this on a 400Gb/s link. If we were to transmit
> the whole batch of LSPs at once, it takes a whopping 20us.  Not
> milliseconds, microseconds.  2x10^-5s.  Clearly, we are not going to be
> rate limited by bandwidth.
>
> Note that 20us is an unreasonable lower bound: we cannot reasonably expect
> a node to absorb 1k PDUs back to back without loss today, in addition to
> all of it’s other responsibilities.
>
> At the opposite end of the spectrum, suppose we transmit one PDU every
> 33ms.  That’s then going to take us 33 seconds to complete. Unreasonably
> slow.
>
> How can we then maximize our goodput?  We know that the receiver has a set
> of buffers and a processing rate that it can support. The processing rate
> will vary, depending on other loads.
>
> What we would like the transmitter to do is to transmit enough to create a
> small processing queue on the receiver and then transmit at the receiver’s
> processing rate.
>
> Can we agree on this goal?
>
> Tony
>
> ___
> Lsr mailing list
> Lsr@ietf.org
> https://www.ietf.org/mailman/listinfo/lsr
>
___
Lsr mailing list
Lsr@ietf.org
https://www.ietf.org/mailman/listinfo/lsr

Re: [Lsr] Dynamic flow control for flooding

[Tony Przygienda]

On Tue, Jul 23, 2019 at 5:24 PM Les Ginsberg (ginsberg) 
wrote:

> Tony –
>
>
>
> As usual, you cover a lot of territory – and even after a couple of
> readings I am not sure I got everything.
>

I was being accused of being too flowerly in my prose for many years so I
adopted an acerbic, terse style ;-)

>
> *From:* Tony Przygienda 
> *Sent:* Tuesday, July 23, 2019 1:56 PM
> *To:* Les Ginsberg (ginsberg) 
> *Cc:* Tony Li ; lsr@ietf.org
> *Subject:* Re: [Lsr] Dynamic flow control for flooding
>
>
>
>
>
>
>
> It is a mistake to equate LSP flooding with a set of independent P2P
> “connections” – each of which can operate at a rate independent of the
> other.
>
>
>
>
>
>
>
> At least my experience much disagrees with that and such a proposal seems
> to steer towards slowest receiver in the whole network problem so I wait
> for others to chime in.
>
> *[Les:] This is NOT AT ALL where I am going.*
>
> *If I have a “large network” and I have a node which consistently cannot
> support the flooding rates necessary to deal with Tony Li’s example (node w
> many neighbors fails) then the network has a problem.*
>
> *Slowing down everyone to meet the flooding speed of the slowest speed is
> not something I would expect a customer to accept. The network will not be
> able to deliver the convergence expected. The node in question needs to be
> identified and steps taken to either fix it or upgrade or replace it or…*
>
>
>
> *The point I am also making is trying to run the network with some links
> flooding fast and some links flooding slow isn’t a solution either.*
>

hmm, then I don't know what you propose in normal case except saying
nothing seems to skin the cat properly when your network is loop-sided
enough. On which we agree I guess ...


>
>
> Then, to clarify on Tony's mail, the "problem" I mentioned anecdotally
> yesterday as behavior I saw on things I did in their time was of course
> when processors were still well under 1GHz and links in Gigs and not 10s
> and 100s of Gigs we have today but yes, the limiting factor was the
> flooding rate (or rather effective processing rate of receiver AFAIR before
> it started drop the RX queues or was late enough to cause RE-TX on senders)
> in terms of losses/retransmissions necessary that were causing transients
> to the point it looked to me then the cure seemed worse than the disease
> (while the disease was likely a flu then compared to today given we didn't
> have massively dense meshes we steer towards today). The base spec &
> mandated flooding numbers didn't change but what is possible in terms of
> rates when breaking the spec did change of course in terms of CPU/links
> speed albeit most ISIS implementations go back to megahertz processors
> still ;-) And the dinner was great BTW ;-)
>
>
>
> So yes, I do think that anything that will flood @ reasonable rate without
> excessive losses will work well on well-computed
> double-flood-reduced-graph, the question is how to get the "reasonable" in
> place both in terms of numbers as well as mechanism for which we saw tons
> lively discussions/proposal yesterday, most obvious being of course going
> and manually bumping e'one's implementation to the desired (? ;-) value
> ...  Other consideration is having computation always trying to get more
> than 2 links in minimal cut on the graph of course which should alleviate
> any bottleneck or rather, make the cut less likely. Given quality of
> max-disjoint-node/link graph computation algorithms that should be doable
> by gut feeling. If e.g. the flood rate per link is available the algorithms
> should be doing even better in centralized case.
>
>
>
> *[Les:] Convergence issues and flooding overload as a result of excessive
> redundant flooding is a real issue – but it is a different problem (for
> which we have solutions) and we should not introduce that issue into this
> discussion.*
>

hmm, we are trying to build flood reduction to deal with exactly this
problem I thought and we are trying to find a good solution in the design
space between a hamiltonian path and not reducing any links @ all where on
one hand the specter of long flooding chains & partitions on single link
failures looms while beckoning with very low CPU load and on the other hand
we can do nothing @ all while staring down the abyss of excessivly large,
densely meshed networks and falling of the cliff of melted flooding ...
So, I'm not sure I introduced anything new but if I did, ignore my attempt
@ clarification of what I said yesterday ...

--- tony

>
___
Lsr mailing list
Lsr@ietf.org
https://www.ietf.org/mailman/listinfo/lsr

Re: [Lsr] Dynamic flow control for flooding

[Les Ginsberg (ginsberg)]

Tony –

As usual, you cover a lot of territory – and even after a couple of readings I 
am not sure I got everything.
Still, I dare to reply.
Inline.

From: Tony Przygienda 
Sent: Tuesday, July 23, 2019 1:56 PM
To: Les Ginsberg (ginsberg) 
Cc: Tony Li ; lsr@ietf.org
Subject: Re: [Lsr] Dynamic flow control for flooding



It is a mistake to equate LSP flooding with a set of independent P2P 
“connections” – each of which can operate at a rate independent of the other.



At least my experience much disagrees with that and such a proposal seems to 
steer towards slowest receiver in the whole network problem so I wait for 
others to chime in.
[Les:] This is NOT AT ALL where I am going.
If I have a “large network” and I have a node which consistently cannot support 
the flooding rates necessary to deal with Tony Li’s example (node w many 
neighbors fails) then the network has a problem.
Slowing down everyone to meet the flooding speed of the slowest speed is not 
something I would expect a customer to accept. The network will not be able to 
deliver the convergence expected. The node in question needs to be identified 
and steps taken to either fix it or upgrade or replace it or…

The point I am also making is trying to run the network with some links 
flooding fast and some links flooding slow isn’t a solution either.

Then, to clarify on Tony's mail, the "problem" I mentioned anecdotally 
yesterday as behavior I saw on things I did in their time was of course when 
processors were still well under 1GHz and links in Gigs and not 10s and 100s of 
Gigs we have today but yes, the limiting factor was the flooding rate (or 
rather effective processing rate of receiver AFAIR before it started drop the 
RX queues or was late enough to cause RE-TX on senders) in terms of 
losses/retransmissions necessary that were causing transients to the point it 
looked to me then the cure seemed worse than the disease (while the disease was 
likely a flu then compared to today given we didn't have massively dense meshes 
we steer towards today). The base spec & mandated flooding numbers didn't 
change but what is possible in terms of rates when breaking the spec did change 
of course in terms of CPU/links speed albeit most ISIS implementations go back 
to megahertz processors still ;-) And the dinner was great BTW ;-)

So yes, I do think that anything that will flood @ reasonable rate without 
excessive losses will work well on well-computed double-flood-reduced-graph, 
the question is how to get the "reasonable" in place both in terms of numbers 
as well as mechanism for which we saw tons lively discussions/proposal 
yesterday, most obvious being of course going and manually bumping e'one's 
implementation to the desired (? ;-) value ...  Other consideration is having 
computation always trying to get more than 2 links in minimal cut on the graph 
of course which should alleviate any bottleneck or rather, make the cut less 
likely. Given quality of max-disjoint-node/link graph computation algorithms 
that should be doable by gut feeling. If e.g. the flood rate per link is 
available the algorithms should be doing even better in centralized case.

[Les:] Convergence issues and flooding overload as a result of excessive 
redundant flooding is a real issue – but it is a different problem (for which 
we have solutions) and we should not introduce that issue into this discussion.

   Les

BTW, with all that experience (MANET did its share in different space as we 
know in terms of flood reduction as well) in RIFT we chose a solution based on 
MANET derivative where every source chooses a different set of trees to flood 
on using Fisher-Yates hashes but that seems possible only if you have 
directionality on the graph (that's what I said once on the mike that doing 
flood reduction in a lattice [partial rank-ordered graph with upper & lower 
bounds] is fairly trivial, on generic graphs not so much necessarily). But 
maybe Pascal reads that and gives it a think ;-)

as usual, 2 cents to improve the internet ;-)

--- tony
___
Lsr mailing list
Lsr@ietf.org
https://www.ietf.org/mailman/listinfo/lsr

[Lsr] Yangdoctors last call review of draft-ietf-ospf-yang-23

[Ladislav Lhotka via Datatracker]

Reviewer: Ladislav Lhotka
Review result: Ready with Nits

I reviewed already revision 09 of this module [1]. All substantial
objections and suggestions expressed in that review are addressed in
revision 23 and I am satisfied with the result. I especially
appreciate that descriptions were considerably expanded and references
added in many places.

I tested validity of the ietf-ospf module with pyang and Yangson
tools, and found no issues. The comments below are non-substantial and
do not affect practical use of the module.

In summary, I think this YANG module and document is a remarkable
piece of work demonstrating that it is possible to build quite complex
vendor-neutral data model that can be used equally well with several
router plaforms.

Comments:

 - names of locally-defined identities as parameters of XPath
   functions derived-from and derived-from-or-self sometimes have
   the 'ospf:' prefix, sometimes don't. I suggest to be
   consistent, and the option without a prefix looks better to me.

 - RFC 8407 suggests this format of references to RFC:
   RFC : Title of the Document
   This draft uses a hyphen instead of a colon. I suggest to
   follow the 8407 convention so as to make parsing easier.

 - the title of Sec. 2.8 should be "OSPF Notifications" (plural
   and capitalization)

 - enumerations "nssa-translator-state-type" and
   "restart-status-type" define the value parameter
   for two of their enums but not for the third. This should be
   avoided.

[1] 
https://datatracker.ietf.org/doc/review-ietf-ospf-yang-09-yangdoctors-lc-lhotka-2017-12-06/
 


___
Lsr mailing list
Lsr@ietf.org
https://www.ietf.org/mailman/listinfo/lsr

Re: [Lsr] Dynamic flow control for flooding

[David Lamparter]

Hi Les,


On Tue, Jul 23, 2019 at 08:29:30PM +, Les Ginsberg (ginsberg) wrote:
> [...] As network-wide convergence depends upon fast propagation of LSP
> changes -

you're losing me between that previous part and the next:

> - which in turn requires consistent flooding rates on all interfaces
> enabled for flooding [...]

I understand and follow your reasoning if we have a classical timer that
limits flooding rates per LSP.  If we get multiple updates to the same
LSP, dissimilar flooding rates imply we might just have sent out the
previous now-outdated state, and we block for some potentially lengthy
time before sending out the most recent version of that LSP.

I don't understand how we get delayed propagation of LSP changes if we
employ some mechanism to raise the flooding rate to something based
around the target system's capabilities.

Could you elaborate on how we get delayed LSP propagation in this
scenario?

Thanks,


-David

___
Lsr mailing list
Lsr@ietf.org
https://www.ietf.org/mailman/listinfo/lsr

Re: [Lsr] Dynamic flow control for flooding

[Tony Przygienda]

>
>
> It is a mistake to equate LSP flooding with a set of independent P2P
> “connections” – each of which can operate at a rate independent of the
> other.
>
>
>
>
At least my experience much disagrees with that and such a proposal seems
to steer towards slowest receiver in the whole network problem so I wait
for others to chime in.

Then, to clarify on Tony's mail, the "problem" I mentioned anecdotally
yesterday as behavior I saw on things I did in their time was of course
when processors were still well under 1GHz and links in Gigs and not 10s
and 100s of Gigs we have today but yes, the limiting factor was the
flooding rate (or rather effective processing rate of receiver AFAIR before
it started drop the RX queues or was late enough to cause RE-TX on senders)
in terms of losses/retransmissions necessary that were causing transients
to the point it looked to me then the cure seemed worse than the disease
(while the disease was likely a flu then compared to today given we didn't
have massively dense meshes we steer towards today). The base spec &
mandated flooding numbers didn't change but what is possible in terms of
rates when breaking the spec did change of course in terms of CPU/links
speed albeit most ISIS implementations go back to megahertz processors
still ;-) And the dinner was great BTW ;-)

So yes, I do think that anything that will flood @ reasonable rate without
excessive losses will work well on well-computed
double-flood-reduced-graph, the question is how to get the "reasonable" in
place both in terms of numbers as well as mechanism for which we saw tons
lively discussions/proposal yesterday, most obvious being of course going
and manually bumping e'one's implementation to the desired (? ;-) value
  Other consideration is having computation always trying to get more
than 2 links in minimal cut on the graph of course which should alleviate
any bottleneck or rather, make the cut less likely. Given quality of
max-disjoint-node/link graph computation algorithms that should be doable
by gut feeling. If e.g. the flood rate per link is available the algorithms
should be doing even better in centralized case.

BTW, with all that experience (MANET did its share in different space as we
know in terms of flood reduction as well) in RIFT we chose a solution based
on MANET derivative where every source chooses a different set of trees to
flood on using Fisher-Yates hashes but that seems possible only if you have
directionality on the graph (that's what I said once on the mike that doing
flood reduction in a lattice [partial rank-ordered graph with upper & lower
bounds] is fairly trivial, on generic graphs not so much necessarily). But
maybe Pascal reads that and gives it a think ;-)

as usual, 2 cents to improve the internet ;-)

--- tony
___
Lsr mailing list
Lsr@ietf.org
https://www.ietf.org/mailman/listinfo/lsr

Re: [Lsr] Dynamic flow control for flooding

[Les Ginsberg (ginsberg)]

Tony –

Thanx for picking up the discussion.
Thanx also for doing the math to show that bandwidth is not a concern. I think 
most/all of us knew that – but it is good to put that small question behind us.

I also think we all agree on the goal - which is to flood significantly faster 
than many implementations do today to handle deployments like the case you 
mention below.

Beyond this point, I have a different perspective.

As network-wide convergence depends upon fast propagation of LSP changes – 
which in turn requires consistent flooding rates on all interfaces enabled for 
flooding – a properly provisioned network MUST be able to sustain a consistent 
flooding rate or the operation of the network will suffer. We therefore need to 
view flow control issues as indicative of a problem.

It is a mistake to equate LSP flooding with a set of independent P2P 
“connections” – each of which can operate at a rate independent of the other.

If we can agree on this, then I believe we will have placed the flow control 
problem in its proper perspective – in which case it will become easier to 
agree on the best way to implement flow control.

   Les



From: Lsr mailto:lsr-boun...@ietf.org>> On Behalf Of Tony 
Li
Sent: Tuesday, July 23, 2019 6:34 AM
To: lsr@ietf.org
Subject: [Lsr] Dynamic flow control for flooding


Hi all,

I’d like to continue the discussion that we left off with last night.

The use case that I posited was a situation where we had 1000 LSPs to flood. 
This is an interesting case that can happen if there was a large network that 
partitioned and has now healed.  All LSPs from the other side of the partition 
are going to need to be updated.

Let’s further suppose that the LSPs have an average size of 1KB.  Thus, the 
entire transfer is around 1MB.

Suppose that we’re doing this on a 400Gb/s link. If we were to transmit the 
whole batch of LSPs at once, it takes a whopping 20us.  Not milliseconds, 
microseconds.  2x10^-5s.  Clearly, we are not going to be rate limited by 
bandwidth.

Note that 20us is an unreasonable lower bound: we cannot reasonably expect a 
node to absorb 1k PDUs back to back without loss today, in addition to all of 
it’s other responsibilities.

At the opposite end of the spectrum, suppose we transmit one PDU every 33ms.  
That’s then going to take us 33 seconds to complete. Unreasonably slow.

How can we then maximize our goodput?  We know that the receiver has a set of 
buffers and a processing rate that it can support. The processing rate will 
vary, depending on other loads.

What we would like the transmitter to do is to transmit enough to create a 
small processing queue on the receiver and then transmit at the receiver’s 
processing rate.

Can we agree on this goal?

Tony

___
Lsr mailing list
Lsr@ietf.org
https://www.ietf.org/mailman/listinfo/lsr

[Lsr] Dynamic flow control for flooding

[Tony Li]

Hi all,

I’d like to continue the discussion that we left off with last night.

The use case that I posited was a situation where we had 1000 LSPs to
flood. This is an interesting case that can happen if there was a large
network that partitioned and has now healed.  All LSPs from the other side
of the partition are going to need to be updated.

Let’s further suppose that the LSPs have an average size of 1KB.  Thus, the
entire transfer is around 1MB.

Suppose that we’re doing this on a 400Gb/s link. If we were to transmit the
whole batch of LSPs at once, it takes a whopping 20us.  Not milliseconds,
microseconds.  2x10^-5s.  Clearly, we are not going to be rate limited by
bandwidth.

Note that 20us is an unreasonable lower bound: we cannot reasonably expect
a node to absorb 1k PDUs back to back without loss today, in addition to
all of it’s other responsibilities.

At the opposite end of the spectrum, suppose we transmit one PDU every
33ms.  That’s then going to take us 33 seconds to complete. Unreasonably
slow.

How can we then maximize our goodput?  We know that the receiver has a set
of buffers and a processing rate that it can support. The processing rate
will vary, depending on other loads.

What we would like the transmitter to do is to transmit enough to create a
small processing queue on the receiver and then transmit at the receiver’s
processing rate.

Can we agree on this goal?

Tony
___
Lsr mailing list
Lsr@ietf.org
https://www.ietf.org/mailman/listinfo/lsr