Re: [PATCH] Update default QoS markers for ssh
Just for the record the latest OpenBSD snapshot with the changed ssh / sshd causes ssh connections to fail across a VMWare Fusion 10 NAT network. ssh into the box you get a banner sent before the connection is closed. ssh out of the like updating a CVS tree gets you... Obtaining OpenBSD-current src tree. packet_write_wait: Connection to 130.102.79.219 port 22: Broken pipe cvs [update aborted]: end of file from server (consult above messages if any) Finished obtaining OpenBSD-current src tree. Adding IPQoS lowdelay to both the ssh_config / ssh_config make this work as before. Machines on the Internal network can SSH into and out amongst each other but going across the NAT boundary to the real world fails. With the state of VMWare (the company) I doubt there will be a fix for Fusion anytime soon. Leaving this here for the archives. Ian McWilliam From: owner-t...@openbsd.org on behalf of Stuart Henderson Sent: Monday, 2 April 2018 12:06 AM To: Job Snijders Cc: tech@openbsd.org Subject: Re: [PATCH] Update default QoS markers for ssh On 2018/04/01 13:29, Job Snijders wrote: > On Sun, Apr 01, 2018 at 11:29:55AM +0100, Stuart Henderson wrote: > > On 2018/03/31 16:10, Job Snijders wrote: > > > TL;DR: I propose to update the defaults to use DSCP "AF21" (Low > > > Latency Data) for interactive session traffic, and CS1 ("Lower > > > Effort") for non-interactive traffic. This applies to both IPv4 and > > > IPv6. > > > > I think this is the right thing to do, but needs handling in > > conjunction with changes to PF, which has dual queue-setting > > (IPTOS_LOWDELAY packets go in one queue along with empty TCP ACKs, > > other packets can go in another lower priority queue). > > > > Since firewalls are often updated less often than hosts I think it > > would be better if the PF side was done first with some time to give > > people chance to update, though it doesn't need to be too long (they > > can set qos in sshd_config if they want to retain the old setting). > > > > A few other pieces use IPTOS_LOWDELAY (pfsync, carp, vxlan, etherip) > > which I think would want switching too. > > What we're looking at is some kind of "migrate TOS to DSCP" project. > There is a few tens of files where TOS values need to be examined and an > appropiate DSCP value choosen. Yes, that's clearly a much bigger scope. But if we're starting with ssh those should be on the radar too IMO. >I'm not sure I'd go as far as blindly > replacing IPTOS_LOWDELAY with IPTOS_DSCP_AF21, but perhaps as a > transition that is what is needed in some places. Indeed. For some of these uses (like carp) something like CS7 seems more appropriate. ssh seems a great place to start gaining more experience with this - in particular any breakage can be handled more easily than things like carp (since ssh already has config options for this). > We can start by treating IPTOS_LOWDELAY and IPTOS_DSCP_AF21 similarly in > pf (see untested patch below), and then accept patches that migrate from > TOS to DSCP? That approach generally makes sense to me. It seems that some other codepoints should probably also be treated as higher priority but I'm not quite sure which yet... > As far as I understand, out-of-the-box pf doesn't do anything with TOS > values, so users wlil have to have explicitly configured something on > the firewall or clients anyhow? Yes, the firewall needs to have queues enabled and traffic assigned with "set queue (foo, bar)" syntax. > Kind regards, > > Job > > share/man/man5/pf.conf.5 | 12 +--- > sys/net/pf.c | 8 +--- > 2 files changed, 14 insertions(+), 6 deletions(-) > > diff --git share/man/man5/pf.conf.5 share/man/man5/pf.conf.5 > index 13e23423daa..f5a65e1b0d4 100644 > --- share/man/man5/pf.conf.5 > +++ share/man/man5/pf.conf.5 > @@ -679,7 +679,9 @@ interface, the queueing priority will also be written as > the priority > code point in the 802.1Q VLAN header. > If two priorities are given, TCP ACKs with no data payload and packets > which have a TOS of > -.Cm lowdelay > +.Cm lowdelay , > +or DSCP value > +.Cm af21 , > will be assigned to the second one. > Packets with a higher priority number are processed first, > and packets with the same priority are processed > @@ -702,7 +704,9 @@ section. > Packets matching this rule will be assigned to the specified > .Ar queue . > If two queues are given, packets which have a TOS of > -.Cm lowdelay > +.Cm lowdelay , > +or DSCP value > +.Cm af21 , > and TCP ACKs with no data payload will be assigned to the second one. > See > .Sx
Re: [PATCH] Update default QoS markers for ssh
On 2018/04/01 08:59, Theo de Raadt wrote: > > I think this is the right thing to do, but needs handling in conjunction > > with changes to PF, which has dual queue-setting (IPTOS_LOWDELAY packets > > pf has to change first?? > > I don't understand the requirement that pf must be capable of handling > this naunce of packets, before any of our applications are changed. > > other applications outside openbsd have already been adapted to use > DSCP. > > there is traffic on the internet doing this, and yet noone has died. Why not? PF has a feature which is *exactly* designed to work with this, specifically so ssh still works when the line is flooded, and the minimum fix is a couple of lines of diff..
Re: [PATCH] Update default QoS markers for ssh
> I think this is the right thing to do, but needs handling in conjunction > with changes to PF, which has dual queue-setting (IPTOS_LOWDELAY packets pf has to change first?? I don't understand the requirement that pf must be capable of handling this naunce of packets, before any of our applications are changed. other applications outside openbsd have already been adapted to use DSCP. there is traffic on the internet doing this, and yet noone has died.
Re: [PATCH] Update default QoS markers for ssh
On 2018/04/01 13:29, Job Snijders wrote: > On Sun, Apr 01, 2018 at 11:29:55AM +0100, Stuart Henderson wrote: > > On 2018/03/31 16:10, Job Snijders wrote: > > > TL;DR: I propose to update the defaults to use DSCP "AF21" (Low > > > Latency Data) for interactive session traffic, and CS1 ("Lower > > > Effort") for non-interactive traffic. This applies to both IPv4 and > > > IPv6. > > > > I think this is the right thing to do, but needs handling in > > conjunction with changes to PF, which has dual queue-setting > > (IPTOS_LOWDELAY packets go in one queue along with empty TCP ACKs, > > other packets can go in another lower priority queue). > > > > Since firewalls are often updated less often than hosts I think it > > would be better if the PF side was done first with some time to give > > people chance to update, though it doesn't need to be too long (they > > can set qos in sshd_config if they want to retain the old setting). > > > > A few other pieces use IPTOS_LOWDELAY (pfsync, carp, vxlan, etherip) > > which I think would want switching too. > > What we're looking at is some kind of "migrate TOS to DSCP" project. > There is a few tens of files where TOS values need to be examined and an > appropiate DSCP value choosen. Yes, that's clearly a much bigger scope. But if we're starting with ssh those should be on the radar too IMO. >I'm not sure I'd go as far as blindly > replacing IPTOS_LOWDELAY with IPTOS_DSCP_AF21, but perhaps as a > transition that is what is needed in some places. Indeed. For some of these uses (like carp) something like CS7 seems more appropriate. ssh seems a great place to start gaining more experience with this - in particular any breakage can be handled more easily than things like carp (since ssh already has config options for this). > We can start by treating IPTOS_LOWDELAY and IPTOS_DSCP_AF21 similarly in > pf (see untested patch below), and then accept patches that migrate from > TOS to DSCP? That approach generally makes sense to me. It seems that some other codepoints should probably also be treated as higher priority but I'm not quite sure which yet... > As far as I understand, out-of-the-box pf doesn't do anything with TOS > values, so users wlil have to have explicitly configured something on > the firewall or clients anyhow? Yes, the firewall needs to have queues enabled and traffic assigned with "set queue (foo, bar)" syntax. > Kind regards, > > Job > > share/man/man5/pf.conf.5 | 12 +--- > sys/net/pf.c | 8 +--- > 2 files changed, 14 insertions(+), 6 deletions(-) > > diff --git share/man/man5/pf.conf.5 share/man/man5/pf.conf.5 > index 13e23423daa..f5a65e1b0d4 100644 > --- share/man/man5/pf.conf.5 > +++ share/man/man5/pf.conf.5 > @@ -679,7 +679,9 @@ interface, the queueing priority will also be written as > the priority > code point in the 802.1Q VLAN header. > If two priorities are given, TCP ACKs with no data payload and packets > which have a TOS of > -.Cm lowdelay > +.Cm lowdelay , > +or DSCP value > +.Cm af21 , > will be assigned to the second one. > Packets with a higher priority number are processed first, > and packets with the same priority are processed > @@ -702,7 +704,9 @@ section. > Packets matching this rule will be assigned to the specified > .Ar queue . > If two queues are given, packets which have a TOS of > -.Cm lowdelay > +.Cm lowdelay , > +or DSCP value > +.Cm af21 , > and TCP ACKs with no data payload will be assigned to the second one. > See > .Sx QUEUEING > @@ -1608,7 +1612,9 @@ Normally only one > .Ar queue > is specified; when a second one is specified it will instead be used for > packets which have a TOS of > -.Cm lowdelay > +.Cm lowdelay , > +or DSCP value > +.Cm af21 , > and for TCP ACKs with no data payload. > .Pp > To continue the previous example, the examples below would specify the > diff --git sys/net/pf.c sys/net/pf.c > index d841f834af1..aac20603a01 100644 > --- sys/net/pf.c > +++ sys/net/pf.c > @@ -2804,7 +2804,7 @@ pf_build_tcp(const struct pf_rule *r, sa_family_t af, > h->ip_len = htons(tlen); > h->ip_v = 4; > h->ip_hl = sizeof(*h) >> 2; > - h->ip_tos = IPTOS_LOWDELAY; > + h->ip_tos = IPTOS_DSCP_AF21; > h->ip_len = htons(len); > h->ip_off = htons(ip_mtudisc ? IP_DF : 0); > h->ip_ttl = ttl ? ttl : ip_defttl; > @@ -7020,7 +7020,8 @@ done: > pf_scrub(pd.m, s->state_flags, pd.af, s->min_ttl, > s->set_tos); > pf_tag_packet(pd.m, s->tag, s->rtableid[pd.didx]); > - if (pqid || (pd.tos & IPTOS_LOWDELAY)) { > + if (pqid || (pd.tos & IPTOS_LOWDELAY) > + || (pd.tos & IPTOS_DSCP_AF21)) { > qid = s->pqid; > if (s->state_flags & PFSTATE_SETPRIO) >
Re: [PATCH] Update default QoS markers for ssh
On Sun, Apr 01, 2018 at 11:29:55AM +0100, Stuart Henderson wrote: > On 2018/03/31 16:10, Job Snijders wrote: > > TL;DR: I propose to update the defaults to use DSCP "AF21" (Low > > Latency Data) for interactive session traffic, and CS1 ("Lower > > Effort") for non-interactive traffic. This applies to both IPv4 and > > IPv6. > > I think this is the right thing to do, but needs handling in > conjunction with changes to PF, which has dual queue-setting > (IPTOS_LOWDELAY packets go in one queue along with empty TCP ACKs, > other packets can go in another lower priority queue). > > Since firewalls are often updated less often than hosts I think it > would be better if the PF side was done first with some time to give > people chance to update, though it doesn't need to be too long (they > can set qos in sshd_config if they want to retain the old setting). > > A few other pieces use IPTOS_LOWDELAY (pfsync, carp, vxlan, etherip) > which I think would want switching too. What we're looking at is some kind of "migrate TOS to DSCP" project. There is a few tens of files where TOS values need to be examined and an appropiate DSCP value choosen. I'm not sure I'd go as far as blindly replacing IPTOS_LOWDELAY with IPTOS_DSCP_AF21, but perhaps as a transition that is what is needed in some places. We can start by treating IPTOS_LOWDELAY and IPTOS_DSCP_AF21 similarly in pf (see untested patch below), and then accept patches that migrate from TOS to DSCP? As far as I understand, out-of-the-box pf doesn't do anything with TOS values, so users wlil have to have explicitly configured something on the firewall or clients anyhow? Kind regards, Job share/man/man5/pf.conf.5 | 12 +--- sys/net/pf.c | 8 +--- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git share/man/man5/pf.conf.5 share/man/man5/pf.conf.5 index 13e23423daa..f5a65e1b0d4 100644 --- share/man/man5/pf.conf.5 +++ share/man/man5/pf.conf.5 @@ -679,7 +679,9 @@ interface, the queueing priority will also be written as the priority code point in the 802.1Q VLAN header. If two priorities are given, TCP ACKs with no data payload and packets which have a TOS of -.Cm lowdelay +.Cm lowdelay , +or DSCP value +.Cm af21 , will be assigned to the second one. Packets with a higher priority number are processed first, and packets with the same priority are processed @@ -702,7 +704,9 @@ section. Packets matching this rule will be assigned to the specified .Ar queue . If two queues are given, packets which have a TOS of -.Cm lowdelay +.Cm lowdelay , +or DSCP value +.Cm af21 , and TCP ACKs with no data payload will be assigned to the second one. See .Sx QUEUEING @@ -1608,7 +1612,9 @@ Normally only one .Ar queue is specified; when a second one is specified it will instead be used for packets which have a TOS of -.Cm lowdelay +.Cm lowdelay , +or DSCP value +.Cm af21 , and for TCP ACKs with no data payload. .Pp To continue the previous example, the examples below would specify the diff --git sys/net/pf.c sys/net/pf.c index d841f834af1..aac20603a01 100644 --- sys/net/pf.c +++ sys/net/pf.c @@ -2804,7 +2804,7 @@ pf_build_tcp(const struct pf_rule *r, sa_family_t af, h->ip_len = htons(tlen); h->ip_v = 4; h->ip_hl = sizeof(*h) >> 2; - h->ip_tos = IPTOS_LOWDELAY; + h->ip_tos = IPTOS_DSCP_AF21; h->ip_len = htons(len); h->ip_off = htons(ip_mtudisc ? IP_DF : 0); h->ip_ttl = ttl ? ttl : ip_defttl; @@ -7020,7 +7020,8 @@ done: pf_scrub(pd.m, s->state_flags, pd.af, s->min_ttl, s->set_tos); pf_tag_packet(pd.m, s->tag, s->rtableid[pd.didx]); - if (pqid || (pd.tos & IPTOS_LOWDELAY)) { + if (pqid || (pd.tos & IPTOS_LOWDELAY) + || (pd.tos & IPTOS_DSCP_AF21)) { qid = s->pqid; if (s->state_flags & PFSTATE_SETPRIO) pd.m->m_pkthdr.pf.prio = s->set_prio[1]; @@ -7032,7 +7033,8 @@ done: } else { pf_scrub(pd.m, r->scrub_flags, pd.af, r->min_ttl, r->set_tos); - if (pqid || (pd.tos & IPTOS_LOWDELAY)) { + if (pqid || (pd.tos & IPTOS_LOWDELAY) + || (pd.tos & IPTOS_DSCP_AF21)) { qid = r->pqid; if (r->scrub_flags & PFSTATE_SETPRIO) pd.m->m_pkthdr.pf.prio = r->set_prio[1];
Re: [PATCH] Update default QoS markers for ssh
On 2018/03/31 16:10, Job Snijders wrote: > TL;DR: I propose to update the defaults to use DSCP "AF21" (Low Latency > Data) for interactive session traffic, and CS1 ("Lower Effort") for > non-interactive traffic. This applies to both IPv4 and IPv6. I think this is the right thing to do, but needs handling in conjunction with changes to PF, which has dual queue-setting (IPTOS_LOWDELAY packets go in one queue along with empty TCP ACKs, other packets can go in another lower priority queue). Since firewalls are often updated less often than hosts I think it would be better if the PF side was done first with some time to give people chance to update, though it doesn't need to be too long (they can set qos in sshd_config if they want to retain the old setting). A few other pieces use IPTOS_LOWDELAY (pfsync, carp, vxlan, etherip) which I think would want switching too. > interactive sessions and the second for non-interactive sessions. > The default is > -.Cm lowdelay > +.Cm af21 > +.Ar (Low-Latency Data) > for interactive sessions and > -.Cm throughput > +.Cm CS1 > +.Ar (Lower Effort) Mismatching caps/lower case (between these two and with the "accepted values" list).
[PATCH] Update default QoS markers for ssh
Dear all, There may be opportunity for improvement of ssh(1) and sshd(8)'s default QoS markers for better integration in environments that can offer either layer-2 or layer-3 prioritisation profiles. Currently ssh(1) and sshd(8) set obsoleted values 'lowdelay' for interactive sessions and 'throughput' for non-interactive sessions. TL;DR: I propose to update the defaults to use DSCP "AF21" (Low Latency Data) for interactive session traffic, and CS1 ("Lower Effort") for non-interactive traffic. This applies to both IPv4 and IPv6. The openbsd 'lowdelay' value translates to IP TOS '0x10', and 'throughput' translates to IP TOS 0x08'. The IP TOS field is 8 bits, and was defined in RFC 1349 (1992 A.D.). IP TOS was deprecated in RFC 2474 (1998 A.D.) with the introduction of "Differentiated Services Field" aka DSCP. When the world transitioned from TOS to DSCP, the IP TOS field was chopped up into two parts: the 6 most significant bits became the DSField (which contains a DSCP), and the 2 least significant bits became a place for ECN experimentation. (Similarly in IPv6, the 'Traffic Class' became the DSField). To convert the IP TOS value to a DSCP value you shift the value 2 bits to the right: 'lowdelay' = DSCP 0x04 and 'throughput' = DSCP 0x02. It should be noted that DSCP 0x04 and DSCP 0x02 have no meaning. Both 0x02 and 0x04 map to Precedence "routine", which is the /least/ important Precedence. Using IP TOS values is problematic because network operators can't match on TOS values on today's equipment. We looked at Cisco IOS XR, Juniper Junos, and Nokia SR-OS - and there aren't classifiers to match on TOS values. Instead, the systems I mentioned allow you to match on DSCP value (and perhaps IPv4 precedence). The ability to correctly match QoS markers of this nature is important in controlled environments, you'll generally see that the DSCP values are fully trusted, or not honored at all (such as when crossing Internet boundaries). An example would be how traffic from VOIP phones/apps are treated in enterprise office networks, perhaps even in the entire branch network. The industry practise is that iff QoS markers are honored, they are translated 1:1 - so AF21 goes into AF21 (or equivalent). I selected AF21 as this is the highest priority within the low-latency service class (and it is higher than what we have today). SSH is elastic and time-sensitive data, where a user is waiting for a response via the network in order to continue with a task at hand. As such, these flows should be considered foreground traffic, with delays or drops to such traffic directly impacting user-productivity. You may ask, "why not EF?" - the risk with EF is that you can easily end up in a tiny policer queue, as an example some CPEs by default only allow up to 100Kbps of EF traffic. EF is meant for inelastic traffic, where for instance the codec sends packets at the rate the codec produces them, regardless of availability of capacity. For bulk SSH traffic, the "Lower Effort" marker was chosen to enable networks implementing a scavanger/lower-than-best effort class to discriminate scp(1) below normal activities, such as web surfing. In general this type of bulk SSH traffic is a background activity. Today this traffic would go into 'best effort', potentially drowning out other traffic on wireless links, so kicking this into 'background' may improve the user experience. An advantage of using "AF21" for interactive SSH and "CS1" for bulk SSH is that they are recognisable values on all common platforms (IANA https://www.iana.org/assignments/dscp-registry/dscp-registry.xml ), and for AF21 specifically a rigorous definition of the intended behavior exists https://tools.ietf.org/html/rfc4594#section-4.7 in addition to the definition of the Assured Forwarding PHB group https://tools.ietf.org/html/rfc2597, and for CS1 (Lower Effort) there is https://tools.ietf.org/html/rfc3662 - finally this document is also a recommened reading: https://tools.ietf.org/html/rfc8325 Another advantage is that the first three bits of "AF21" map to the equivalent I 802.1D PCP, IEEE 802.11e, MPLS EXP/Cos and IP Precedence value of 2 (also known as "Immediate", or "AC_BE"), and CS1's first 3 bits map to I 802.1D PCP, IEEE 802.11e, MPLS/Cos and IP Precedence value 1 ("Background" or "AC_BK"). The below patch ensures that in environments where people are using QoS, the openssh defaults can easily matched both for interactive and bulk SSH traffic in all layers of the transport stack (wired/wirless Ethernet, MPLS, IP) and priortize accordingly. Kind regards, Job The below patch should be portable, but I've only tested it on my own openbsd machines. usr.bin/ssh/readconf.c| 4 ++-- usr.bin/ssh/servconf.c| 4 ++-- usr.bin/ssh/ssh_config.5 | 6 -- usr.bin/ssh/sshd_config.5 | 6 -- 4 files changed, 12 insertions(+), 8 deletions(-) diff --git usr.bin/ssh/readconf.c usr.bin/ssh/readconf.c index 5d17b725600..d3a121373d8 100644 --- usr.bi