Thanks, Willy. That's exactly the kind of analysis I was trying to elicit.
Comments in line.
On 5/2/2022 8:23 PM, Willy Tarreau wrote:
On Mon, May 02, 2022 at 02:27:15PM -0700, Paul Vixie wrote:
a partially managed (not fully transparent) network (public or private) can
be expected to implement port-based inbound UDP blocking of the kind you
describe. the set of ports will be dynamic, updated during attacks. not
something to be hardcoded or "set and forget".
I think that's the most important aspect, and for having participated many
times to fightng DDoS I agree with this. There are obvious common sanity
rules that are often applied based on what is possible/available:
- have a (set of) local DNS and NTP server(s) that are the only ones
susceptible of receiving UDP from privilege ports (< 1024) and have
all other servers use DNS and NTP from there
- have the edge components (routers, L3 switches) block all inbound
UDP from privileged and well-known source ports to regular servers;
this list may evolve during attacks
- have all servers locally refine their blocking list at the kernel
level (netfilter, BPF, PF and so on etc) preventing cross-protocol
communication (53->123, 123->53 etc).
- when possible, refine filtering by source/destination ports based
on the expected protocol (e.g. verify that traffic to port 53 looks
like a DNS request, from port 53 looks like a DNS response, from
123 looks like an NTP response, with special cases for 53->53 or
123->123)
If the wrong packet is delivered to the userspace application, you lose
anyway as most of the harm caused by network stack traversal by a packet
was already done. Worse once a packet passes through, it's often trivial
for the attacker to repeat it and flood the application. A good rule of
thumb is to count on 1 million packets per second delivered to the
application, per CPU core. Of course it will vary a lot between systems
and the filtering in place, but the order of magnitude is there. This
means that a 100G NIC can keep 148 cores busy just delivering bad
packets to be dropped by the application.
As I said, there appear to be two concerns. One is to protect the local
server from floods, and, as you say, this is much better done by
blocking the suspicious traffic ahead of the server. The other is,
provide at least a modicum of defense against attacks that use the QUIC
server as a reflector. For example:
* attacker sends a "carefully crafted" UDP packet to QUIC server,
spoofing the source address and port of a target,
* QUIC server processes the packet and produces the expected "response"
* target attempts to process the packet and something bad happens.
The concern here is not so much volumetric attacks, which can be
mitigated with a combination of not amplifying traffic and carefully
using retry. We are also concerned about "Request Forgery" attacks, in
which the "response" activates a bug in the target. For example, QUIC
sends a long packet to a poorly written game server, and the game server
crashes when trying to process that packet. There may also be variants
of that attack in which an attacking server bounces packets through a
QUIC client.
Clearly, servers that can be crashed by a few magic packets have no
business being reachable from the Internet. But the concern here is that
by spoofing the source address, an attacker located outside of the
server network can reflect and attack towards a vulnerable server inside
that network, even if firewalls isolate that vulnerable server from the
Internet.
I would suggest that port filtering at the application layer is only
used to decide whether to respond or not (i.e. should I send a retry
for a packet that parses correctly). For example it could be suggested
that packets coming from suspicious but valid source ports ought to
be double-checked with a retry to preserve local resources. But that
will not be used to save CPU anyway.
Yes. Makes sense. But there are four ways by which QUIC can bounce packets:
* Initial packets from the client will cause the server to send a flight
of up to 3 Handshake packets. The Retry mechanism could be used there.
* Initial packets from the client with an unsupported version number
cause the server to send a Version Negotiation packet. These are short
packets, but up to 256 bytes of the VN packet content can be
predetermined by the attacker.
* 1RTT packets sent to a server with a unknown Connection ID may cause
the server to send a Stateless Reset packet. The Stateless Reset packet
must be shorter than the incoming packet, so there is no amplification.
The content is hard to predict, but who knows.
* After a connection has been established and verified, either party can
send a "path validation" packet from a "new" address and port. Clients
or servers will respond with a path validation packet of their own,
trying to reach the new address. The validation attempt could be
repeated multiple time, typically 3 times. At least 20 bytes at the
beginning of the packet can be controlled by the attacker, and possibly
many more.
If an application just drops packets from a suspicious port, it
mitigates all 4 avenues of attack. If it want precision instead, it has
to write specific code at four locations. RFC 9000 details these
attacks, but mostly says "protect using ingress filtering, per BCP38".
Maintaining a public list of well-known suspicious ports can be both
helpful and dangerous. It's helpful in that it shows implementers that
some ranges should never appear as they're normally reserved for
listening services, that some numbers are well known and widely
deployed, and that some are less common but appear anywhere, indicating
that a range is not sufficient. This should help design a flexible
filtering mechanism. But it can also be dangerous if applied as-is:
this list cannot remain static as new entries may have to be added
within a few minutes to hours hours and each addition will cause extra
breakage, thus some previous ones will likely be dropped after a
previous service stopped being widely attacked. I.e. the list in
question likely needs to be accessible by configuration in field and
not be hard-coded.
AFAIK, such lists are hard coded in quite a few implementations. Even if
they are provided in a configuration file, changing the file requires
restarting the server. So "within a few minutes" may be problematic.
Other approaches could work fairly well, such as keeping a rate counter
of incoming packets per source port (e.g. unparsable or initial packets)
which, above a moderate value, would serve to send retry, and above a
critical value, can be used to decide to block the port (possibly earlier
in the chain). This remains reasonably cheap to implement, though it may
end up causing some flapping as the rate will fall after the port is
blocked upstream, which may lead to it being reopened before the attack
is finished.
Yes. Should that really be "per source port" or "per source IP + port"?
I think we'll discover new fun things over time and will learn new
attacks and workarounds as deployments grow.
I have not heard of actual attacks "in the wild", but who knows...
-- Christian Huitema
