Re: Broken PMTUD / ICMP blackhole?
On Thu, 19 Dec 2019 12:35:06 -0600 John Hasler wrote: > Celejar writes: > > ...the problem only occurs when tethering. > > I wrote: > > Which is the only time the cellular encapsulation is being done. > > Celejar writes: > > Understood. I had been responding to your point about the wifi > > encapsulation. > > You've eliminated that by demonstrating that the problem only occurs > when cellular is involved. > > >>From here the requirements of cellular encapsulation look a lot like the > requirements for DSL encapsulation and so the solutions may be similar. > Worst case, they may even be the same. Thanks. I don't know anything about DSL encapsulation, and it sounds like I don't really want to ;) Celejar
Re: Broken PMTUD / ICMP blackhole?
Celejar writes: > ...the problem only occurs when tethering. I wrote: > Which is the only time the cellular encapsulation is being done. Celejar writes: > Understood. I had been responding to your point about the wifi > encapsulation. You've eliminated that by demonstrating that the problem only occurs when cellular is involved. >From here the requirements of cellular encapsulation look a lot like the requirements for DSL encapsulation and so the solutions may be similar. Worst case, they may even be the same. -- John Hasler jhas...@newsguy.com Elmwood, WI USA
Re: Broken PMTUD / ICMP blackhole?
On Wed, 18 Dec 2019 21:46:29 -0600 John Hasler wrote: > Celejar writes: > > ...the problem only occurs when tethering. > > Which is the only time the cellular encapsulation is being done. Understood. I had been responding to your point about the wifi encapsulation. Celejar
Re: Broken PMTUD / ICMP blackhole?
Celejar writes: > ...the problem only occurs when tethering. Which is the only time the cellular encapsulation is being done. -- John Hasler jhas...@newsguy.com Elmwood, WI USA
Re: Broken PMTUD / ICMP blackhole?
On Wed, 18 Dec 2019 13:58:38 -0600 John Hasler wrote: > Celejar writes: > > I assume the phone first just routes them from wifi to cellular. I'm > > not familiar with how it then transmits them over the cellular link. > > It has to encapsulate them in some way for the cellular protocol. > > > Yes, but that happens with virtually all this machine's network > > connections. > > I thought you said it only happened when using tethering. I suppose I misunderstood you. I meant that the wifi encapsulation takes place with virtually all the machine's network connections, since it is rarely connected via wired ethernet, and that this encapsulation was therefore not likely the cause of the problem, since the problem only occurs when tethering. Celejar
Re: Broken PMTUD / ICMP blackhole?
Celejar writes: > I assume the phone first just routes them from wifi to cellular. I'm > not familiar with how it then transmits them over the cellular link. It has to encapsulate them in some way for the cellular protocol. > Yes, but that happens with virtually all this machine's network > connections. I thought you said it only happened when using tethering. -- John Hasler jhas...@newsguy.com Elmwood, WI USA
Re: Broken PMTUD / ICMP blackhole?
On Wed, 18 Dec 2019 07:29:08 -0600 John Hasler wrote: > Celejar writes: > > I'm not that familiar with the internals, but basically, the phone > > presents a wifi access point, the computer connects to it as it would > > to any AP, and the phone apparently routes packets to and from the > > cellular network. > > Yes, I know that. However, the packets are being encapsulated in some > way. They may be encapsulating the IP packets directly or they may be > encapsulating the ethernet packets as PPP does. Either way there is an I assume the phone first just routes them from wifi to cellular. I'm not familiar with how it then transmits them over the cellular link. > opportunity for problems similar to those that have developed with > PPPoE because encapsulization always involves adding headers. The > packets are being encapsulated for the WiFi too, of course. Yes, but that happens with virtually all this machine's network connections. Celejar
Re: Broken PMTUD / ICMP blackhole?
On Wed, 18 Dec 2019 07:47:18 -0600 John Hasler wrote: > Celejar writes: > > It does work (and to automate it, I'll probably put it in e/n/i with a > > post-up line) - I'm just looking for the "right" way to do it (and to > > undo it on connection down, as I discussed in another message in this > > thread). > > I wouldn't worry about doing it the *right* way. It's a kludge that > should not be necessary but is. > > Automatic adjustment would be interesting but I don't see an obvious > method. Fair enough. Celejar
Re: Broken PMTUD / ICMP blackhole?
Celejar writes: > It does work (and to automate it, I'll probably put it in e/n/i with a > post-up line) - I'm just looking for the "right" way to do it (and to > undo it on connection down, as I discussed in another message in this > thread). I wouldn't worry about doing it the *right* way. It's a kludge that should not be necessary but is. Automatic adjustment would be interesting but I don't see an obvious method. -- John Hasler jhas...@newsguy.com Elmwood, WI USA
Re: Broken PMTUD / ICMP blackhole?
Celejar writes: > I'm not that familiar with the internals, but basically, the phone > presents a wifi access point, the computer connects to it as it would > to any AP, and the phone apparently routes packets to and from the > cellular network. Yes, I know that. However, the packets are being encapsulated in some way. They may be encapsulating the IP packets directly or they may be encapsulating the ethernet packets as PPP does. Either way there is an opportunity for problems similar to those that have developed with PPPoE because encapsulization always involves adding headers. The packets are being encapsulated for the WiFi too, of course. -- John Hasler jhas...@newsguy.com Elmwood, WI USA
Re: Broken PMTUD / ICMP blackhole?
On Wed, 18 Dec 2019 10:59:11 - (UTC) Curt wrote: > On 2019-12-18, Anthony DeRobertis wrote: > > > > Another option might be to use Network Manager, I think its connections > > can set a custom MTU, but I'm not 100% sure as I've never tried it. > > I've used tethering successfully with Network Manager but never had > occasion to alter the mtu settings. > > Why would wouldn't > > ip link set dev mtu 1300 > > work? It does work (and to automate it, I'll probably put it in e/n/i with a post-up line) - I'm just looking for the "right" way to do it (and to undo it on connection down, as I discussed in another message in this thread). Celejar
Re: Broken PMTUD / ICMP blackhole?
On Wed, 18 Dec 2019 02:30:01 -0500 Anthony DeRobertis wrote: > On 12/17/19 11:39 AM, Celejar wrote: > > > > Now I just have have to figure out the best place to configure this. > > I'm using dhcp via /etc/network/interfaces, but the 'dhcp' method > > doesn't seem to support manual MTU setting. I could use a 'supersede > > interface-mtu' line in dhclient.conf, but AFAICT, options there apply to > > all dhcp connections, and I can't make out a simple way to set them on > > a per connection basis. I suppose I could always just put 'ip link set > > wlan0 mtu 1440' in a script and hook it in to the appropriate > > 'iface' stanza in e/n/i with a 'post-up' line. > > Personally, I'd just use the post-up line. I'll probably do that - but I'm not sure how to restore the proper MTU on connection down (since I'm not sure that my other connections are setting it at all, as opposed to just leaving it as is). I suppose I can just assume that it should always be 1500, so I can just use a pre/post-down line to set it to 1500 on connection down, but I wonder if there's a better way. I suppose I could also save the current MTU to some file somewhere when I lower the MTU, and then restore it afterward ... > Ideally whichever device is the DHCP server would set the appropriate > MTU via its DHCP response, but if that's the phone, I have no idea how > you'd fix that easily. (That's why there is a supersede setting for it, Well, it's LineageOS, a fairly open OS, so I can probably figure out how to hack it, if necessary. > it can be part of the DHCP response — that's how different MTUs from > different DHCP connections should work.) > > Another option might be to use Network Manager, I think its connections > can set a custom MTU, but I'm not 100% sure as I've never tried it. > > BTW: dhclient.conf options can be per-interface (via an interface > section), so your wifi and wired adapters could be different. I occasionally use wired, but I'm almost always on wifi. Thanks, Celejar
Re: Broken PMTUD / ICMP blackhole?
On 2019-12-18, Anthony DeRobertis wrote: > > Another option might be to use Network Manager, I think its connections > can set a custom MTU, but I'm not 100% sure as I've never tried it. I've used tethering successfully with Network Manager but never had occasion to alter the mtu settings. Why would wouldn't ip link set dev mtu 1300 work? > BTW: dhclient.conf options can be per-interface (via an interface > section), so your wifi and wired adapters could be different. > > -- "J'ai pour me guérir du jugement des autres toute la distance qui me sépare de moi." Antonin Artaud
Re: Broken PMTUD / ICMP blackhole?
On 12/17/19 11:39 AM, Celejar wrote: Now I just have have to figure out the best place to configure this. I'm using dhcp via /etc/network/interfaces, but the 'dhcp' method doesn't seem to support manual MTU setting. I could use a 'supersede interface-mtu' line in dhclient.conf, but AFAICT, options there apply to all dhcp connections, and I can't make out a simple way to set them on a per connection basis. I suppose I could always just put 'ip link set wlan0 mtu 1440' in a script and hook it in to the appropriate 'iface' stanza in e/n/i with a 'post-up' line. Personally, I'd just use the post-up line. Ideally whichever device is the DHCP server would set the appropriate MTU via its DHCP response, but if that's the phone, I have no idea how you'd fix that easily. (That's why there is a supersede setting for it, it can be part of the DHCP response — that's how different MTUs from different DHCP connections should work.) Another option might be to use Network Manager, I think its connections can set a custom MTU, but I'm not 100% sure as I've never tried it. BTW: dhclient.conf options can be per-interface (via an interface section), so your wifi and wired adapters could be different.
Re: Broken PMTUD / ICMP blackhole?
On Tue, 17 Dec 2019 13:38:59 -0600 John Hasler wrote: > Celejar writes: > > ...on my normal network connection, with > > the MTU left at the normal 1500, I have no problems. > > What *kind* of connection? Verizon Fios residential. But I have used my system with dozens of different internet connections, and I do not usually experience these sorts of problems. > > It's only while on this particular connection via cell phone tether > > that I see problems. > > The problem I experienced had to do with pppoe overhead and fixed packet Yes, many of the discussions I have seen of needing to lower MTU refer to PPPoE. > size but I see no reason a similar problem couldn't crop up elsewhere. > How does the tethering work? I'm not that familiar with the internals, but basically, the phone presents a wifi access point, the computer connects to it as it would to any AP, and the phone apparently routes packets to and from the cellular network. Celejar
Re: Broken PMTUD / ICMP blackhole?
Celejar writes: > ...on my normal network connection, with > the MTU left at the normal 1500, I have no problems. What *kind* of connection? > It's only while on this particular connection via cell phone tether > that I see problems. The problem I experienced had to do with pppoe overhead and fixed packet size but I see no reason a similar problem couldn't crop up elsewhere. How does the tethering work? -- John Hasler jhas...@newsguy.com Elmwood, WI USA
Re: Broken PMTUD / ICMP blackhole?
On Tue, 17 Dec 2019 12:53:21 -0600 John Hasler wrote: > tomas writes: > > I don't know the error message by heart, but here, it seems > > the message size is too big for your local MTU... > > Celejar writes: > > Yes, I think this is pretty clear. The local wifi interface has the > > standard MTU of 1500, so it rejects packets larger than that. > > > With = 1472, I get, at least sometimes: > > From 192.168.43.245 icmp_seq=2 Frag needed and DF set (mtu = 1472) > > tomas writes: > > This is definitely an ICMP message you receive from some upstream > > Celejar writes: > > Yes, except that I don't see this message consistently. I assume that's > > some sort of upstream flakiness. > > It has to do with TLS. Recent changes in the protocol have had the > result that it sometimes sends packets too large for the standard MTUs. > These packets cannot be fragmented, so you get intermittent problems > that seem like they must be at the other end. I've had to reduce my MTU > to 1300. I understand (sort of) why TLS is triggering the problem, but it's fundamentally a PMTUD problem: on my normal network connection, with the MTU left at the normal 1500, I have no problems. It's only while on this particular connection via cell phone tether that I see problems. Celejar
Re: Broken PMTUD / ICMP blackhole?
tomas writes: > I don't know the error message by heart, but here, it seems > the message size is too big for your local MTU... Celejar writes: > Yes, I think this is pretty clear. The local wifi interface has the > standard MTU of 1500, so it rejects packets larger than that. > With = 1472, I get, at least sometimes: > From 192.168.43.245 icmp_seq=2 Frag needed and DF set (mtu = 1472) tomas writes: > This is definitely an ICMP message you receive from some upstream Celejar writes: > Yes, except that I don't see this message consistently. I assume that's > some sort of upstream flakiness. It has to do with TLS. Recent changes in the protocol have had the result that it sometimes sends packets too large for the standard MTUs. These packets cannot be fragmented, so you get intermittent problems that seem like they must be at the other end. I've had to reduce my MTU to 1300. -- John Hasler jhas...@newsguy.com Elmwood, WI USA
Re: Broken PMTUD / ICMP blackhole?
On Tue, 17 Dec 2019 07:15:21 +0100 wrote: > On Mon, Dec 16, 2019 at 10:37:12PM -0500, Celejar wrote: > > Hi, > > > > I have a Debian Sid system with generally working networking. Recently, > > I experienced some strange connectivity problems with a particular > > network connection [...] > > > PING 1.1.1.1 (1.1.1.1) 1492(1520) bytes of data. > > ping: local error: message too long, mtu=1500 > > I don't know the error message by heart, but here, it seems > the message size is too big for your local MTU... Yes, I think this is pretty clear. The local wifi interface has the standard MTU of 1500, so it rejects packets larger than that. > > With = 1472, I get, at least sometimes: > > > > >From 192.168.43.245 icmp_seq=2 Frag needed and DF set (mtu = 1472) > > This is definitely an ICMP message you receive from some upstream Yes, except that I don't see this message consistently. I assume that's some sort of upstream flakiness. > > followed by (for various values of ): > > > > ping: local error: message too long, mtu=1472 > > > > until I drop below 1444, at which point I once again get no reply, > > until <= 1412, at which point I once again get normal ping replies. > > Someone upstream is dropping the packets, perhaps sending an ICMP > back (possibly "fragmentation needed"), perhaps someone else is Yes, that's what's supposed to happen, according to everything I've read, but overly aggressive blocking of ICMP is apparently a common problem. > dropping that ICMP (your firewall, perhaps?) Not running one on the local machine, and I'm pretty sure that my gateway firewall, an OpenWrt installation in a fairly standard configuration, isn't configuring to do that (and don't forget that I do get some "Frag needed" messages, as above). > > For comparison purposes, on a normal, properly behaving network > > connection, I get normal ping replies for <= 1472, and "message > > too long" for > 1472. > > > > Am I understanding this correctly, that there's some kind of PMTUD / > > ICMP blackhole problem here? > > This would be my interpretation too. > > > If so, what can I do about it? My > > understanding is that I can either set the MTU lower on the client, or > > do MSS clamping. Any suggestions? Is this something Mint / T-Mobile, or > > someone upstream, is just messing up? > > Since you're not getting the ICMPs back, your only choice seems to be > to reduce your MTU, manually yes. Okay. Since the largest packets that get through consistently are 1440 bytes: $ ping -M do 1.1.1.1 -s 1412 PING 1.1.1.1 (1.1.1.1) 1412(1440) bytes of data. 1420 bytes from 1.1.1.1: icmp_seq=1 ttl=55 time=103 ms I tried (manually) setting the MTU to 1440. So far I have indeed seen no further problems. Now I just have have to figure out the best place to configure this. I'm using dhcp via /etc/network/interfaces, but the 'dhcp' method doesn't seem to support manual MTU setting. I could use a 'supersede interface-mtu' line in dhclient.conf, but AFAICT, options there apply to all dhcp connections, and I can't make out a simple way to set them on a per connection basis. I suppose I could always just put 'ip link set wlan0 mtu 1440' in a script and hook it in to the appropriate 'iface' stanza in e/n/i with a 'post-up' line. Thanks for the help, Celejar
Re: Broken PMTUD / ICMP blackhole?
On Mon, Dec 16, 2019 at 10:37:12PM -0500, Celejar wrote: > Hi, > > I have a Debian Sid system with generally working networking. Recently, > I experienced some strange connectivity problems with a particular > network connection [...] > PING 1.1.1.1 (1.1.1.1) 1492(1520) bytes of data. > ping: local error: message too long, mtu=1500 I don't know the error message by heart, but here, it seems the message size is too big for your local MTU... > With = 1472, I get, at least sometimes: > > >From 192.168.43.245 icmp_seq=2 Frag needed and DF set (mtu = 1472) This is definitely an ICMP message you receive from some upstream > followed by (for various values of ): > > ping: local error: message too long, mtu=1472 > > until I drop below 1444, at which point I once again get no reply, > until <= 1412, at which point I once again get normal ping replies. Someone upstream is dropping the packets, perhaps sending an ICMP back (possibly "fragmentation needed"), perhaps someone else is dropping that ICMP (your firewall, perhaps?) > For comparison purposes, on a normal, properly behaving network > connection, I get normal ping replies for <= 1472, and "message > too long" for > 1472. > > Am I understanding this correctly, that there's some kind of PMTUD / > ICMP blackhole problem here? This would be my interpretation too. > If so, what can I do about it? My > understanding is that I can either set the MTU lower on the client, or > do MSS clamping. Any suggestions? Is this something Mint / T-Mobile, or > someone upstream, is just messing up? Since you're not getting the ICMPs back, your only choice seems to be to reduce your MTU, manually yes. Cheers -- t signature.asc Description: Digital signature
