On 5/3/2022 11:53 PM, Willy Tarreau wrote:
On Tue, May 03, 2022 at 11:00:11PM -0700, Christian Huitema wrote:
QUIC includes a feature to facilitate moving a connection from an anycast
address to a unicast address. The client finds the anycast address through
the DNS, contacts the server, learns a "preferred server address" during the
handshake, and tries validating that preferred address before using it for
the rest of the connection duration, without risk losing the connection to a
routing event that redirects anycast to a different server. Which is nice,
but of course a villainous server could feed a bogus preferred address.
That's the client bounce for you. All the server packets before the bounce
are sent with the expected 5 tuple. And of course the "preferred address" is
sent using encrypted packets.
Ah, I wasn't aware, seen like this, that's fun. Well, I guess the client
will try to connect to the new address with some info allowing to identify
the origin server (e.g. SNI), thus allowing a QUIC decoder/server at that
address to spot where the client found the address, no ?
Everything is encrypted, except for a few bytes in the packet header
that carry a connection identifier, i.e., an octet string chosen by the
server. The server provides the client with a list of those; if needed,
that list is known by the load balancer at the server side, so the LB
can forward the packet to the right server in the farm. The encryption
key is specific to the connection, so only the right server can respond.
There *will* inevitably be some problems at the beginning, but one good
thing is that solutions are reasonably simple, can instantly be deployed
in field using port filtering, and will quickly land in the code itself
because the various stack authors have full control of their code.
Port filtering at the edge will not catch the "preferred address attack" in
which the server proposes a non-anycast address to the client, so I guess
QUIC stacks will have to protect that. As in, if a server says "please move
the connection to 10.0.0.1:53", the client should just decline and stick to
the anycast address. And yes, using a configurable list makes sense.
Yes, that's a good point, I agree. I think quite a bit of the burden will
be on the clients' shoulder. It reminds me the good joke we had in the 90s,
creating a "warez.<domain>" DNS entry pointing to 127.0.0.1. I've personally
seen someone trapped and saying "what, they stole'my data!" after ftping
there :-)
Yes. Should that really be "per source port" or "per source IP + port"?
Per port should be sufficient. There's no reason from the server side that
multiple clients collectively decide to use the same source port to send
legit traffic. And using the port alone will allow to quicky detect a new
internet-wide attack coming from a randomly vulnerable service (IoT
gateways, ToIP etc) regardless of their address. I don't know how this
would be distributed, but I guess that if a source port is responsible
for 100 times more traffic than all other one combined, it will usually
be a good indicator that you don't want to deal with that traffic anymore.
I would expect quite a bit of legit server-to-server traffic on ports 443 or
853,
I think that's exactly the thing we must avoid. Encouraging (or even
allowing) this is exactly what will make it impossible for infrastructure
to defend itself against massive attacks. There's no value in using the
same port on both sides. The savings of sockets and source ports are
already enormous, going from N to 1 from TCP to QUIC, if an agent needs
to make outgoing connections, it certainly can afford a second socket,
which is nothing compared to the hundreds of thousands it would have used
in TCP. Not being willing to do that is putting the internet at risk. For
example among the protections you usually find at the edge is the blocking
of a TCP SYN packet from a source port below 1024. A few services in the
past (rlogin, rsh) used to bind to ports 512-1023 to "prove" that the
user ID was authenticated by the local OS, but that's never seen over the
net. One service has caused trouble to this, active FTP with source port
20, and rules which were a bit too lax used to allow it to connect to any
service (e.g. SMTP) thereby allowing to use FTP servers to send spams by
retrieving files that contained SMTP protocol contents. This has
contributed to making active FTP unpopular, and nowadays it has become
safe to block SYN from sources < 1024 at the edge. UDP doesn't have such
a thing as a SYN flag and it's critical that traffic cannot be made
symmetrical, or there's no more infrastructure filtering and only
application level filtering.
Too bad that nobody discussed that during the review of
https://datatracker.ietf.org/doc/draft-ietf-quic-manageability/, that
could have been a nice place for such considerations.
It seems that we really need to write all these recommendations in a
draft. Services that are peer-to-peer in nature would tend to use a
single port.Same for QUIC stacks. It is a little bit easier to use a
single socket than to force outgoing connections on a single socket. If
we want them to use two separate ports, we have to explain convincingly.
On the other hand, I think we are past the "port below 1024" rules. For
example, memcached is on port 11211, MDNS on port 5353, and there are
many more like that.
but yes, keeping a dynamic list of most abused ports makes sense.
Although that's mostly an issue for volumetric attacks, and host based
defenses are not too efficient there, but still it feels better than just
receiving the attack and waiting until it stops.
Definitely.
I think we'll discover new fun things over time and will learn new
attacks and workarounds as deployments grow.
I have not heard of actual attacks "in the wild", but who knows...
It's too early, stacks are still evolving and the level of deployment
is not high yet. That doesn't mean nobody is trying to elaborate new
attacks. For example yesterday on my home servers which do not advertise
QUIC I received one short packet from port 443 to port 443 from a host
named "scanners.labs.something", and a 1238-byte one from a random port
to 443 from a host named "scan-19a.somethingelse". Thus it shows that
studies are already in progress. We just need to be patient :-)
We probably will not have to wait too long...
I hope so. It's important to get real-world feedback on protocols early
enough to allow them to improve. It's especially important right now that
any QUIC site can fall back to H1/H2 over TCP so it's easy to just block
it in case of massive failure. If big problems are reported only 10 years
after deployment it will be much harder to disable it in emergency.
Yes, but out of our control...
-- Christian Huitema
