On Fri, Mar 03, 2023 at 04:04:37PM +0100, Amaury Denoyelle wrote:
> > Can anyone say sth. about client port allocation in haproxy? Is it done
> > manually in some cases? Or is that a task that is completely done by the OS?
>
> To my knowledge, haproxy does not explicitely select the port when
> connecting to a backend server unless a specific "source" statement is
> used, so this should be the responsibility of the OS. Have you checked
> that your ephemeral port range is big enough ?
>
> $ sysctl net.ipv4.ip_local_port_range net.ipv4.ip_local_reserved_ports
Indeed, source ports are chosen by the operating system. The problem
Jack reports is a well-known issue in NAT+multi-homed environments and
is purely an architectural one:
client 1.2.3.4:5678
|
|
/~~~~~~~~~~~~\
{ internet }
`~~~~~~~~~~~~'
/ \
/ \
ISP A ISP B
| |
(X) (X)
\ /
2.2.2.2:443 \ / 3.3.3.3:443
+-------------+
| DNAT |
+-------------+
|
server 4.4.4.4:443
The DNAT box that receives traffic from both ISPs needs to store
a session in its table that contains this:
ext src,dst int src,dst
1.2.3.4:5678 2.2.2.2:443 1.2.3.4:5678 4.4.4.4:443
This way when the server responds to 1.2.3.4:5678, the NAT box reads this
NAT table in reverse direction and finds that 4.4.4.4:443->1.2.3.4:5678
must be translated to 2.2.2.2:443->1.2.3.4:5678.
But what if the same source arrives on the other public IP ? you'll get
a new entry:
ext src,dst int src,dst
1.2.3.4:5678 3.3.3.3:443 1.2.3.4:5678 4.4.4.4:443
and suddenly your NAT table contains a conflict for the reverse lookup
needed for responses:
ext src,dst int src,dst
1.2.3.4:5678 2.2.2.2:443 1.2.3.4:5678 4.4.4.4:443
1.2.3.4:5678 3.3.3.3:443 1.2.3.4:5678 4.4.4.4:443
which one to pick for 4.4.4.4:443->1.2.3.4:5678 ?
In order to avoid this, some NAT boxes support to implement a
double-nat mechanism for incoming traffic. They'll either set a
specific source per original destination, or will assign different
port ranges, so you could have this:
ext src,dst int src,dst
1.2.3.4:5678 2.2.2.2:443 4.4.4.2:5678 4.4.4.4:443
1.2.3.4:5678 3.3.3.3:443 4.4.4.3:5678 4.4.4.4:443
The problem is that the server doesn't receive the original IPs
anymore, which can be a problem for filtering or even legal
logging. In this case you'll need to rely on the NAT box's
logging, or it should implement a mechanism to pass the source
address via an auxiliary mechanism such as the PROXY protocol.
A much simpler method for the device is in fact to assign two
IP addresses to the server and configure the DNAT box to use a
distinct IP address for each ISP:
client 1.2.3.4:5678
|
|
/~~~~~~~~~~~~\
{ internet }
`~~~~~~~~~~~~'
/ \
/ \
ISP A ISP B
| |
(X) (X)
\ /
2.2.2.2:443 \ / 3.3.3.3:443
+-------------+
| DNAT |
+-------------+
| |
4.4.4.4:443 | | 4.4.4.5:443
server
This way there is no more ambiguity in the DNAT table:
ext src,dst int src,dst
1.2.3.4:5678 2.2.2.2:443 1.2.3.4:5678 4.4.4.4:443
1.2.3.4:5678 3.3.3.3:443 1.2.3.4:5678 4.4.4.5:443
If the server responds from 4.4.4.4 then the outgoing source
is necessarily 2.2.2.2 but if it responds from 4.4.4.5, then
the outgoing source is necessarily 3.3.3.3.
If for any reason you cannot fix this deployment this way and
can only act on the haproxy side (e.g. the server is at a
customer's who doesn't want to fix their broken setup and who
decided that it was your problem only), then as Amaury suggests,
the "source" directive allows you to enforce source port ranges
for outgoing traffic. Then you can just have two sets of non-
overlapping ports for each target address:
server isp-a 2.2.2.2:443 source 0.0.0.0:20000-29999
server isp-b 3.3.3.3:443 source 0.0.0.0:30000-39999
That way there will never be any conflict in the fortigate's table
and the return traffic will always find a unique path.
But if the setup is yours, I strongly encourage you to fix it as
explained above because any other client could face this problem.
And in some mobile environments which make use of CGN (carrier-grade
NAT), it is possible that a mobile client will use a very narrow
source port range and will frequently reuse the same ports, which
can cause exactly this problem to happen when connecting to that
site.
Hoping this helps,
Willy
