Re: [GROW] ixp jumbo frames doc

[john heasley]

Thu, Nov 17, 2011 at 04:23:03PM -0800, Martin J. Levy:
> Hence ... There is MSS negotiation; hence there is "value"; but I believe I 
> should only note this fact within the document and not make in any way a 
> requirement to tune or change the BGP or TCP knobs to take advantage of the 
> large MTU path.  The fact that some sessions enable this and some don't is 
> perfectly OK. It's being decided by the routers for whatever reason they 
> choose. 

enable what?  it's TCP that is concerned with the MTU (& MSS), not BGP.
a vendor might provide data about the underlying tcp connection in its
bgp session output, but that doesn't mean that BGP is concerned with its
value - its just handy debugging information for the operator.  I suppose
an implementation could use the MTU or MSS for something, but it shouldn't
need to do that if its TCP stack were implemented properly - implementation
specifc.
___
GROW mailing list
GROW@ietf.org
https://www.ietf.org/mailman/listinfo/grow

Re: [GROW] ixp jumbo frames doc

[Templin, Fred L]

Hi Chris, 

> -Original Message-
> From: Christopher Morrow [mailto:christopher.mor...@gmail.com] 
> Sent: Thursday, November 17, 2011 9:48 PM
> To: Templin, Fred L
> Cc: grow-cha...@tools.ietf.org; grow@ietf.org grow@ietf.org; 
> Martin J. Levy
> Subject: Re: [GROW] ixp jumbo frames doc
> 
> On Thu, Nov 17, 2011 at 1:08 PM, Templin, Fred L
>  wrote:
> > Hi Chris,
> >
> > See below for some follow-up:
> >
> >> -Original Message-
> >> From: grow-boun...@ietf.org [mailto:grow-boun...@ietf.org] On
> >> Behalf Of Christopher Morrow
> >> Sent: Thursday, November 17, 2011 12:09 AM
> >> To: grow-cha...@tools.ietf.org; grow@ietf.org grow@ietf.org;
> >> Martin J. Levy
> >> Subject: [GROW] ixp jumbo frames doc
> >>
> >> We had a bit of a lively discussion in the meeting today about:
> >>   <http://tools.ietf.org/html/draft-mlevy-ixp-jumboframes-00>
> >>
> >> Some of the topics covered were:
> >>   o Is there a problem with CRC problems with 9k ?
> >
> > Please note that the concern is specifically for CRC-32.
> > Several works have discussed the error characteristics
> > of CRC-32 in relation to data set sizes:
> >
> >    Koopman, P., "32-Bit Cyclic Redundancy Codes for
> >    Internet Applications", Dec. 2002.
> >
> >    Stone, J. & Partridge, C., "When the CRC and TCP
> >    Checksum Disagree", Aug. 2000.
> >
> >    Jain, R., "Error Characteristics of Fiber Distributed
> >    Data Interface (FDDI), August 1990.
> >
> > Those works tend to support the assertion that CRC-32
> > is adequate for data set sizes up to "about 9k or even
> > a little bit more".
> >
> > That said, I can easily imagine something stronger than
> > CRC-32, and it is the responsibility of the link layer
> > to provide adequate error checking if it intends to
> > support larger MTUs.
> >
> >>   o is this something that the IEEE reigns supreme on?
> >
> > It is the link layer's job to ensure that it performs
> > adequate error checking for packets of various sizes;
> > it is not the network layer's place to guess at how well
> > the link layer is doing its job. So, the link layer's
> > advertised MTU is a certification that suitable error
> > detection will be performed for packets no larger than
> > the MTU; otherwise, it will become known as a "bad link".
> >
> >>   o should there be other methods of deployment?
> >
> > Other methods of deployment than what?
> >
> 
> Sorry, I was rushing :( One of the proposals was to use a secondary
> VLAN at the exchange for 'larger' packets, one was to forklift the
> exchange, I believe there was a third discussed in the meeting as
> well.

Thanks for the clarification; I have no further comment
on this.

> >>   o is this a BCP instead of Informational doc?
> >
> > The document is proposing jacking up the Internet cell
> > size from 1500 to 9000, where 9000 seems like a good
> > fit for the vast majority of link paths anticipated
> > for the near future. That seems like BCP to me.
> 
> Err, it's proposing setting the IXP interfaces to 'larger than 1500'.
> One of the problems is that some networks already show up at exchanges
> with 'larger than 1500 on their backbone/core facing interfaces, and
> likely on most other interfaces in the network as well, so these IXP
> interfaces are one-off situations :(

Right, but it is asking for a specific minimum MTU of
9000 in the core and is hence striving for a new
Internet cell size. IMHO, this is a necessary first
step toward migration to larger MTUs.

Don't get me wrong, though - the Internet will still
need to accommodate MTU diversity since there is no
way to legislate a sweeping change of this nature
that can be deployed all at once. That means PMTUD
will still be necessary, and should be made as robust
as possible. 
 
> I don't see this proposal as jacking up the whole of the Internet,
> there are still a vast number of edge interfaces which are not capable
> of changing MTU, but making the one-off problems go away seems like a
> good idea.

Right; there are certainly edge interfaces for which
larger MTUs are not practical. Most wireless devices
fall under that category. However, migrating the core
to 9000+ allows edge devices that can do better than
1500 to discover larger MTUs.

> > At the same time, at some point down the road we may
> > need to turn the crank again and jack up from 9000
> > to something still large

Re: [GROW] ixp jumbo frames doc

[Christopher Morrow]

On Thu, Nov 17, 2011 at 1:08 PM, Templin, Fred L
 wrote:
> Hi Chris,
>
> See below for some follow-up:
>
>> -Original Message-
>> From: grow-boun...@ietf.org [mailto:grow-boun...@ietf.org] On
>> Behalf Of Christopher Morrow
>> Sent: Thursday, November 17, 2011 12:09 AM
>> To: grow-cha...@tools.ietf.org; grow@ietf.org grow@ietf.org;
>> Martin J. Levy
>> Subject: [GROW] ixp jumbo frames doc
>>
>> We had a bit of a lively discussion in the meeting today about:
>>   <http://tools.ietf.org/html/draft-mlevy-ixp-jumboframes-00>
>>
>> Some of the topics covered were:
>>   o Is there a problem with CRC problems with 9k ?
>
> Please note that the concern is specifically for CRC-32.
> Several works have discussed the error characteristics
> of CRC-32 in relation to data set sizes:
>
>    Koopman, P., "32-Bit Cyclic Redundancy Codes for
>    Internet Applications", Dec. 2002.
>
>    Stone, J. & Partridge, C., "When the CRC and TCP
>    Checksum Disagree", Aug. 2000.
>
>    Jain, R., "Error Characteristics of Fiber Distributed
>    Data Interface (FDDI), August 1990.
>
> Those works tend to support the assertion that CRC-32
> is adequate for data set sizes up to "about 9k or even
> a little bit more".
>
> That said, I can easily imagine something stronger than
> CRC-32, and it is the responsibility of the link layer
> to provide adequate error checking if it intends to
> support larger MTUs.
>
>>   o is this something that the IEEE reigns supreme on?
>
> It is the link layer's job to ensure that it performs
> adequate error checking for packets of various sizes;
> it is not the network layer's place to guess at how well
> the link layer is doing its job. So, the link layer's
> advertised MTU is a certification that suitable error
> detection will be performed for packets no larger than
> the MTU; otherwise, it will become known as a "bad link".
>
>>   o should there be other methods of deployment?
>
> Other methods of deployment than what?
>

Sorry, I was rushing :( One of the proposals was to use a secondary
VLAN at the exchange for 'larger' packets, one was to forklift the
exchange, I believe there was a third discussed in the meeting as
well.

>>   o is this a BCP instead of Informational doc?
>
> The document is proposing jacking up the Internet cell
> size from 1500 to 9000, where 9000 seems like a good
> fit for the vast majority of link paths anticipated
> for the near future. That seems like BCP to me.

Err, it's proposing setting the IXP interfaces to 'larger than 1500'.
One of the problems is that some networks already show up at exchanges
with 'larger than 1500 on their backbone/core facing interfaces, and
likely on most other interfaces in the network as well, so these IXP
interfaces are one-off situations :(

I don't see this proposal as jacking up the whole of the Internet,
there are still a vast number of edge interfaces which are not capable
of changing MTU, but making the one-off problems go away seems like a
good idea.

> At the same time, at some point down the road we may
> need to turn the crank again and jack up from 9000
> to something still larger. Hence, the BCP for the
> near term should leave open the door for an update
> at some point in the future.
>
>>   o should this be adopted for WG work?
>
> Yes.

thanks!

-chris

> Thanks - Fred
> fred.l.temp...@boeing.com
>
>> We'll pass along a separate call for adoption as well, but keep these
>> points in mind (and hopefully let's chat some about these as well)
>>
>> -chris
>> (co-chair)
>> ___
>> GROW mailing list
>> GROW@ietf.org
>> https://www.ietf.org/mailman/listinfo/grow
>>
>
___
GROW mailing list
GROW@ietf.org
https://www.ietf.org/mailman/listinfo/grow

Re: [GROW] ixp jumbo frames doc

[Martin J. Levy]

I've been waiting for the feedback (which has been great) before commenting or 
editing up a revision to the document; however this is one point that's worth 
noting.

We did a test on a Jumbo Frame enabled IX (NETNOD). We looked at all the peers 
and listed the MSS against the Jumbo or non-Jumbo VLAN. By doing a simple "show 
ip bgp nei" and "show ipv6 bgp nei" command, collecting the data and looking at 
it; I find:

MSS IP ADDRESS  HARDWARE  VLAN
-     

1240***.***.**.189  Cisco 1500
...
1440***.***.**.26   Cisco 1500
1420:***:*:**::26   Cisco 1500
...
1440***.***.**.34   Cisco 4470
1420:***:*:**::34   Cisco 4470
...
1460***.***.**.172  Brocade   4470
...
4410:***:*:**::19   Juniper   4470
4430***.***.**.19   Juniper   4470
...
4410:***:*:**::24   Juniper   4470
4430***.***.**.24   Juniper   4470
...
4430***.***.**.143  Cisco 4470
4430***.***.**.181  Juniper   4470

I only included the interesting peers. There's tons of peers within the 1,4xx 
MSS range; but there are ONLY six with ~4,400 Byte MSS.

The hardware column is the hardware of the peer (we run Brocade).

Hence ... There is MSS negotiation; hence there is "value"; but I believe I 
should only note this fact within the document and not make in any way a 
requirement to tune or change the BGP or TCP knobs to take advantage of the 
large MTU path.  The fact that some sessions enable this and some don't is 
perfectly OK. It's being decided by the routers for whatever reason they 
choose. 

NETNOD's Jumbo Frame setup has been working for many many years (over various 
h/w platforms) and has stood the test of time.

Does this help?

Martin

PS: I'll respond to other points in a bit.


On Nov 17, 2011, at 3:55 PM, Tony Li wrote:

> 
> On Nov 17, 2011, at 3:51 PM, john heasley wrote:
> 
>> Thu, Nov 17, 2011 at 06:46:12PM -0500, Jakob Heitz:
>>> BGP uses TCP. Differing MTU has no impact on update generation in BGP. BGP 
>>> today has a maximum message size of 4096 bytes. TCP slices and dices that 
>>> into IP packets of its own choosing.
>> 
>> i was about to reply with the same, then it occured to me that an
>> implementation may have closer ties to the underlying mtu for rfc2385
>> or similar.  so the question becomes where implementation-specific
>> limitation belong in the draft.
> 
> 
> They don't.
> 
> Tony
> 
> ___
> GROW mailing list
> GROW@ietf.org
> https://www.ietf.org/mailman/listinfo/grow

___
GROW mailing list
GROW@ietf.org
https://www.ietf.org/mailman/listinfo/grow

Re: [GROW] ixp jumbo frames doc

[Tony Li]

On Nov 17, 2011, at 3:51 PM, john heasley wrote:

> Thu, Nov 17, 2011 at 06:46:12PM -0500, Jakob Heitz:
>> BGP uses TCP. Differing MTU has no impact on update generation in BGP. BGP 
>> today has a maximum message size of 4096 bytes. TCP slices and dices that 
>> into IP packets of its own choosing.
> 
> i was about to reply with the same, then it occured to me that an
> implementation may have closer ties to the underlying mtu for rfc2385
> or similar.  so the question becomes where implementation-specific
> limitation belong in the draft.


They don't.

Tony

___
GROW mailing list
GROW@ietf.org
https://www.ietf.org/mailman/listinfo/grow

Re: [GROW] ixp jumbo frames doc

[john heasley]

Thu, Nov 17, 2011 at 06:46:12PM -0500, Jakob Heitz:
> BGP uses TCP. Differing MTU has no impact on update generation in BGP. BGP 
> today has a maximum message size of 4096 bytes. TCP slices and dices that 
> into IP packets of its own choosing.

i was about to reply with the same, then it occured to me that an
implementation may have closer ties to the underlying mtu for rfc2385
or similar.  so the question becomes where implementation-specific
limitation belong in the draft.
___
GROW mailing list
GROW@ietf.org
https://www.ietf.org/mailman/listinfo/grow

Re: [GROW] ixp jumbo frames doc

[Jakob Heitz]

BGP uses TCP. Differing MTU has no impact on update generation in BGP. BGP 
today has a maximum message size of 4096 bytes. TCP slices and dices that into 
IP packets of its own choosing.

--
Jakob Heitz.


On Nov 17, 2011, at 12:37 AM, "iLya"  wrote:

> I've read the draft but couldn't attend the session due to clash in 
> schedule.
> 
> One point which is currently missing in the draft and which I consider very 
> important is effect of transition to "jumbo" on BGP. Many BGP 
> implementations today group updates to similar peers and format them only 
> once then replicate to multiple peers. Formatting is arguably more 
> resource-consuming compare to replication. When MTU is the same for all IXP 
> customers, then most of UPDATE messages will be formatted only once. During 
> transition period there will be peers with different MTU, and depending on 
> BGP implementation more than one formatting will be required. It would be 
> nice if the draft could be updated to address this subject.
> 
> Kind regards,
> iLya
> 
> --
> From: "Christopher Morrow" 
> Sent: Thursday, November 17, 2011 9:09 AM
> To: ; ; "Martin J. Levy" 
> 
> Subject: [GROW] ixp jumbo frames doc
> 
>> We had a bit of a lively discussion in the meeting today about:
>> <http://tools.ietf.org/html/draft-mlevy-ixp-jumboframes-00>
>> 
>> Some of the topics covered were:
>> o Is there a problem with CRC problems with 9k ?
>> o is this something that the IEEE reigns supreme on?
>> o should there be other methods of deployment?
>> o is this a BCP instead of Informational doc?
>> o should this be adopted for WG work?
>> 
>> We'll pass along a separate call for adoption as well, but keep these
>> points in mind (and hopefully let's chat some about these as well)
>> 
>> -chris
>> (co-chair)
>> ___
>> GROW mailing list
>> GROW@ietf.org
>> https://www.ietf.org/mailman/listinfo/grow
>> 
>> 
> 
> 
> 
> ___
> GROW mailing list
> GROW@ietf.org
> https://www.ietf.org/mailman/listinfo/grow
___
GROW mailing list
GROW@ietf.org
https://www.ietf.org/mailman/listinfo/grow

Re: [GROW] ixp jumbo frames doc

[Tony Li]

On Nov 17, 2011, at 10:08 AM, Templin, Fred L wrote:

>>  o is this a BCP instead of Informational doc?
> 
> The document is proposing jacking up the Internet cell
> size from 1500 to 9000, where 9000 seems like a good
> fit for the vast majority of link paths anticipated
> for the near future. That seems like BCP to me.


It should again be noted that the IETF lacks an enforcement arm or any legal 
standing to dictate operations whatsoever.

Thus, even as a BCP, this basically has the effect of saying "we think this is 
a really good idea, would you please do this?"

Tony


___
GROW mailing list
GROW@ietf.org
https://www.ietf.org/mailman/listinfo/grow

Re: [GROW] ixp jumbo frames doc

[Templin, Fred L]

Hi Chris,

See below for some follow-up: 

> -Original Message-
> From: grow-boun...@ietf.org [mailto:grow-boun...@ietf.org] On 
> Behalf Of Christopher Morrow
> Sent: Thursday, November 17, 2011 12:09 AM
> To: grow-cha...@tools.ietf.org; grow@ietf.org grow@ietf.org; 
> Martin J. Levy
> Subject: [GROW] ixp jumbo frames doc
> 
> We had a bit of a lively discussion in the meeting today about:
>   <http://tools.ietf.org/html/draft-mlevy-ixp-jumboframes-00>
> 
> Some of the topics covered were:
>   o Is there a problem with CRC problems with 9k ?

Please note that the concern is specifically for CRC-32.
Several works have discussed the error characteristics
of CRC-32 in relation to data set sizes:

Koopman, P., "32-Bit Cyclic Redundancy Codes for
Internet Applications", Dec. 2002.

Stone, J. & Partridge, C., "When the CRC and TCP
Checksum Disagree", Aug. 2000.

Jain, R., "Error Characteristics of Fiber Distributed
Data Interface (FDDI), August 1990.

Those works tend to support the assertion that CRC-32
is adequate for data set sizes up to "about 9k or even
a little bit more".

That said, I can easily imagine something stronger than
CRC-32, and it is the responsibility of the link layer
to provide adequate error checking if it intends to
support larger MTUs. 

>   o is this something that the IEEE reigns supreme on?

It is the link layer's job to ensure that it performs
adequate error checking for packets of various sizes;
it is not the network layer's place to guess at how well
the link layer is doing its job. So, the link layer's
advertised MTU is a certification that suitable error
detection will be performed for packets no larger than
the MTU; otherwise, it will become known as a "bad link".

>   o should there be other methods of deployment?

Other methods of deployment than what?

>   o is this a BCP instead of Informational doc?

The document is proposing jacking up the Internet cell
size from 1500 to 9000, where 9000 seems like a good
fit for the vast majority of link paths anticipated
for the near future. That seems like BCP to me.

At the same time, at some point down the road we may
need to turn the crank again and jack up from 9000
to something still larger. Hence, the BCP for the
near term should leave open the door for an update
at some point in the future.

>   o should this be adopted for WG work?

Yes.

Thanks - Fred
fred.l.temp...@boeing.com

> We'll pass along a separate call for adoption as well, but keep these
> points in mind (and hopefully let's chat some about these as well)
> 
> -chris
> (co-chair)
> ___
> GROW mailing list
> GROW@ietf.org
> https://www.ietf.org/mailman/listinfo/grow
> 
___
GROW mailing list
GROW@ietf.org
https://www.ietf.org/mailman/listinfo/grow

Re: [GROW] ixp jumbo frames doc

[Christopher Morrow]

On Thu, Nov 17, 2011 at 3:37 AM, iLya  wrote:
> I've read the draft but couldn't attend the session due to clash in
> schedule.
>
> One point which is currently missing in the draft and which I consider very
> important is effect of transition to "jumbo" on BGP. Many BGP
> implementations today group updates to similar peers and format them only
> once then replicate to multiple peers. Formatting is arguably more
> resource-consuming compare to replication. When MTU is the same for all IXP
> customers, then most of UPDATE messages will be formatted only once. During
> transition period there will be peers with different MTU, and depending on
> BGP implementation more than one formatting will be required. It would be
> nice if the draft could be updated to address this subject.

I had heard from John Scudder (and Dave Ward actually) at some point
in the near past that essentially the idea of 'peer-groups' is just
syntactic sugar at this point in time. There is some attempt to keep
like-peers together in update groups, but over time drift in state
happens and peers are separated from the herd.

In the end, this isn't super important in at least 2 bgp
implementations in the field today...

-chris

>
> Kind regards,
> iLya
>
> --
> From: "Christopher Morrow" 
> Sent: Thursday, November 17, 2011 9:09 AM
> To: ; ; "Martin J. Levy"
> 
> Subject: [GROW] ixp jumbo frames doc
>
>> We had a bit of a lively discussion in the meeting today about:
>>  <http://tools.ietf.org/html/draft-mlevy-ixp-jumboframes-00>
>>
>> Some of the topics covered were:
>>  o Is there a problem with CRC problems with 9k ?
>>  o is this something that the IEEE reigns supreme on?
>>  o should there be other methods of deployment?
>>  o is this a BCP instead of Informational doc?
>>  o should this be adopted for WG work?
>>
>> We'll pass along a separate call for adoption as well, but keep these
>> points in mind (and hopefully let's chat some about these as well)
>>
>> -chris
>> (co-chair)
>> ___
>> GROW mailing list
>> GROW@ietf.org
>> https://www.ietf.org/mailman/listinfo/grow
>>
>>
>
>
>
>
___
GROW mailing list
GROW@ietf.org
https://www.ietf.org/mailman/listinfo/grow

Re: [GROW] ixp jumbo frames doc

[iLya]

I've read the draft but couldn't attend the session due to clash in 
schedule.


One point which is currently missing in the draft and which I consider very 
important is effect of transition to "jumbo" on BGP. Many BGP 
implementations today group updates to similar peers and format them only 
once then replicate to multiple peers. Formatting is arguably more 
resource-consuming compare to replication. When MTU is the same for all IXP 
customers, then most of UPDATE messages will be formatted only once. During 
transition period there will be peers with different MTU, and depending on 
BGP implementation more than one formatting will be required. It would be 
nice if the draft could be updated to address this subject.


Kind regards,
iLya

--
From: "Christopher Morrow" 
Sent: Thursday, November 17, 2011 9:09 AM
To: ; ; "Martin J. Levy" 


Subject: [GROW] ixp jumbo frames doc


We had a bit of a lively discussion in the meeting today about:
 <http://tools.ietf.org/html/draft-mlevy-ixp-jumboframes-00>

Some of the topics covered were:
 o Is there a problem with CRC problems with 9k ?
 o is this something that the IEEE reigns supreme on?
 o should there be other methods of deployment?
 o is this a BCP instead of Informational doc?
 o should this be adopted for WG work?

We'll pass along a separate call for adoption as well, but keep these
points in mind (and hopefully let's chat some about these as well)

-chris
(co-chair)
___
GROW mailing list
GROW@ietf.org
https://www.ietf.org/mailman/listinfo/grow






___
GROW mailing list
GROW@ietf.org
https://www.ietf.org/mailman/listinfo/grow

[GROW] ixp jumbo frames doc

[Christopher Morrow]

We had a bit of a lively discussion in the meeting today about:
  

Some of the topics covered were:
  o Is there a problem with CRC problems with 9k ?
  o is this something that the IEEE reigns supreme on?
  o should there be other methods of deployment?
  o is this a BCP instead of Informational doc?
  o should this be adopted for WG work?

We'll pass along a separate call for adoption as well, but keep these
points in mind (and hopefully let's chat some about these as well)

-chris
(co-chair)
___
GROW mailing list
GROW@ietf.org
https://www.ietf.org/mailman/listinfo/grow