Public bug reported:
[Impact]
On Ubuntu 26.04, adding individual IP addresses as sources to a firewalld zone
scales quadratically in both time and memory. Past roughly 130 sources,
firewalld no longer answers within the D-Bus reply timeout and every subsequent
firewall-cmd invocation fails with "Did not receive a reply".
Allow-listing a /24 worth of hosts is an entirely ordinary administrative task,
so in practice source-based zone configuration is unusable on 26.04.
This is upstream bug firewalld/firewalld#1399. Fedora, which ships firewalld
2.5.1, does not reproduce the problem at all: it adds 192 sources in 26 seconds
with a flat ~45 MB memory footprint. The request here is to update / patch the
firewalld package in Ubuntu.
[Steps to reproduce]
time for i in `seq 1 192`; do
echo -n "$i: "
firewall-cmd --zone=trusted --add-source=192.168.1.$i/32
(( i % 10 == 0 )) || continue
date
systemctl status firewalld.service | grep Memory
done
[Expected result]
All sources are added successfully, with time and memory consumption roughly
linear in the number of sources - as observed on Fedora with firewalld 2.5.1.
[Actual result on Ubuntu 26.04]
Memory and per-operation latency both grow super-linearly:
sources firewalld RSS (peak) time for the preceding 10 additions
------- -------------------- -----------------------------------
10 34.1 M (53.8 M) -
20 47.4 M (53.8 M) 2 s
30 67.7 M (69.4 M) 4 s
40 93.5 M (95.8 M) 5 s
50 125.4 M (128.6 M) 7 s
60 168.8 M (176.1 M) 11 s
70 208.6 M (216.4 M) 18 s
80 257.2 M (264.9 M) 30 s
90 324.6 M (340.4 M) 49 s
100 382.5 M (398.3 M) 77 s
110 450.8 M (466.5 M) 105 s
120 535.5 M (559.1 M) 149 s
130 609.7 M (633.4 M) 215 s
140 464.0 M (705.6 M) 281 s
150 793.7 M (825.4 M) 377 s
That is roughly 4.8 MB of RSS per added source over the pre-failure range, and
by source 130 a single --add-source call takes about 21 seconds - just under
the 25 s default D-Bus reply timeout.
From source 132 onward every call fails:
131: success
132: Error: Did not receive a reply. Possible causes include: the remote
application did not send a reply, the message bus security policy
blocked the reply, the reply timeout expired, or the network connection
was broken.
133: Error: Did not receive a reply. ...
(identical error for every source from 132 to 155)
The run was aborted manually at source 155, so there is no final "real" timing
line. Elapsed time can be derived from the timestamps above: sources 10 to 150
took 22 minutes 10 seconds (16:58:04 -> 17:20:14 UTC), and the failure mode was
already unambiguous by that point.
Two further observations:
1. The daemon keeps working after the client gives up. At source 140 the
reported RSS had dropped from 609.7 M to 464.0 M while the peak had risen
from 633.4 M to 705.6 M - the service was not restarted, so this is
in-process allocation and release, not a crash. This means that when
firewall-cmd returns "Did not receive a reply", the administrator cannot
tell whether the rule was applied or not. The exit status of firewall-cmd
no longer reflects the resulting firewall state.
2. Memory is never released back to a sane baseline while the sources remain
configured. RSS peaked at 825.4 M for what amounts to 150 /32 addresses.
[Result on Fedora with firewalld 2.5.1]
The identical script, run to the full 192 sources:
191: success
192: success
real 0m25.688s
user 0m15.722s
sys 0m3.109s
Every one of the 192 additions succeeded. Memory stayed essentially flat and
each batch of ten took one to two seconds throughout, with no degradation as
the source count grew.
[Side-by-side comparison]
Same script, same source addresses, same checkpoints:
sources Ubuntu 26.04 Fedora / firewalld 2.5.1
RSS (peak) per 10 RSS (peak) per 10
------- ----------- ------- ----------- -------
10 34.1 (53.8) - 32.6 (33.1) -
20 47.4 (53.8) 2 s 33.3 (34.1) 1 s
30 67.7 (69.4) 4 s 34.0 (35.1) 2 s
40 93.5 (95.8) 5 s 34.9 (35.1) 1 s
50 125.4 (128.6) 7 s 36.4 (36.4) 1 s
60 168.8 (176.1) 11 s 36.1 (37.4) 1 s
70 208.6 (216.4) 18 s 37.5 (37.8) 2 s
80 257.2 (264.9) 30 s 37.9 (38.7) 1 s
90 324.6 (340.4) 49 s 39.1 (39.3) 1 s
100 382.5 (398.3) 77 s 39.2 (39.7) 1 s
110 450.8 (466.5) 105 s 40.4 (41.2) 2 s
120 535.5 (559.1) 149 s 41.5 (41.7) 1 s
130 609.7 (633.4) 215 s 41.6 (42.6) 1 s
140 464.0 (705.6) 281 s 42.9 (44.2) 2 s
150 793.7 (825.4) 377 s 43.1 (44.2) 1 s
160 (already failing) 43.2 (44.2) 2 s
170 (already failing) 44.0 (44.9) 1 s
180 (already failing) 45.2 (46.0) 2 s
190 (already failing) 45.1 (46.4) 1 s
metric Ubuntu 26.04 Fedora 45
----------------------------- ------------- ------------
memory growth per source ~4.8 MB ~0.07 MB
RSS at 130 sources 609.7 M 41.6 M
time for sources 121-130 215 s 1 s
first failure source 132 none
192 sources completed no yes, 25.7 s
At 130 sources Ubuntu uses roughly 15 times the memory of Fedora, and the
same batch of ten additions takes over 200 times as long. Fedora's cost per
source is flat; Ubuntu's is not.
[Upstream reference]
https://github.com/firewalld/firewalld/issues/1399
Upstream identifies the cause as
_interface_or_source_update_policies_derived_from_zone generating
filter_FORWARD_POLICIES and nat_POSTROUTING_POLICIES entries for every
combination of interfaces and sources, so the generated nftables ruleset grows
quadratically with the number of entries. Rule counts reported upstream
(81,063 rules on firewalld 2.0.1 vs 875 on 0.9.3) match the memory and latency
growth measured here.
The Fedora results above show that firewalld 2.5.1 is not affected.
[Request]
Please update the firewalld package in Ubuntu 26.04 to 2.5.1 or later, or
cherry-pick the upstream fix as an SRU.
[Package information]
Ubuntu 26.04 (affected):
$ lsb_release -rd
Description: Ubuntu 26.04 LTS
Release: 26.04
$ apt-cache policy firewalld
firewalld:
Installed: 2.3.1-3
Candidate: 2.3.1-3
Version table:
*** 2.3.1-3 500
500 http://jp.archive.ubuntu.com/ubuntu resolute/universe amd64
Packages
100 /var/lib/dpkg/status
$ apt-cache policy nftables python3-nftables
nftables:
Installed: 1.1.6-1
Candidate: 1.1.6-1
Version table:
*** 1.1.6-1 500
500 http://jp.archive.ubuntu.com/ubuntu resolute/main amd64 Packages
100 /var/lib/dpkg/status
python3-nftables:
Installed: 1.1.6-1
Candidate: 1.1.6-1
Version table:
*** 1.1.6-1 500
500 http://jp.archive.ubuntu.com/ubuntu resolute/universe amd64
Packages
100 /var/lib/dpkg/status
Architecture: amd64
Deployment: VM
Fedora (not affected):
$ cat /etc/fedora-release
Fedora release 46 (Rawhide)
$ rpm -q firewalld
firewalld-2.5.1-1.fc45.noarch
ProblemType: Bug
DistroRelease: Ubuntu 26.04
Package: firewalld 2.3.1-3
ProcVersionSignature: Ubuntu 7.0.0-30.30-generic 7.0.12
Uname: Linux 7.0.0-30-generic x86_64
ApportVersion: 2.34.1-0ubuntu0.1
Architecture: amd64
CasperMD5CheckResult: pass
Date: Tue Aug 25 17:27:00 2026
InstallationDate: Installed on 2026-06-08 (79 days ago)
InstallationMedia: Ubuntu-Server 26.04 "Resolute Raccoon" - Release amd64
(20260420.1)
PackageArchitecture: all
ProcEnviron:
LANG=C.UTF-8
PATH=(custom, no user)
SHELL=/bin/bash
TERM=xterm
SourcePackage: firewalld
UpgradeStatus: No upgrade log present (probably fresh install)
** Affects: firewalld (Ubuntu)
Importance: Undecided
Status: New
** Tags: amd64 apport-bug resolute
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2165053
Title:
firewalld: adding sources to a zone scales quadratically - 825 MB RSS
and D-Bus timeouts after ~130 sources (fixed in upstream 2.5.1)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/firewalld/+bug/2165053/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
