Public bug reported:
Subject: apcupsd NIS server deadlock: write_nbytes() has no send timeout,
mutex held across network I/O → FD exhaustion under broken TCP
Package: apcupsd
Version: 3.14.14-7 (and all prior versions — bug is in upstream source)
Severity: important
════════════════════════════════════════════════════════════════
SUMMARY
════════════════════════════════════════════════════════════════
The apcupsd NIS server (NISPORT 3551) has two interacting bugs that together
cause file descriptor exhaustion and eventual daemon failure when a TCP client
stops consuming data (e.g. a broken ESTABLISHED connection with no RST/FIN,
as occurs after NAT table expiry or a firewall change):
Bug 1 — src/lib/apclibnis.c: write_nbytes() has no send timeout.
read_nbytes() uses select() with a 15-second timeout; write_nbytes()
calls send() with no guard at all. A peer whose receive buffer is
full causes send() to block for the kernel TCP retry period
(~15 minutes on Linux).
Bug 2 — src/apcnis.c: status_close() holds the global NIS mutex for the
entire duration of sending a status response over the network.
With Bug 1 active, one blocked thread holds the mutex for ~15
minutes. Every other incoming connection creates a new thread
that immediately blocks on the mutex — holding its accepted socket
FD. Threads accumulate until RLIMIT_NOFILE (default 1024) is
exhausted, at which point apcupsd can no longer open any file,
including /etc/hosts.deny, and logs:
apcupsd: cannot open /etc/hosts.deny: Too many open files
OBSERVED SYMPTOM (PRODUCTION)
On a Linux host running apcupsd 3.14.14-7 with UPSTYPE=net:
• Open file count climbs steadily over ~5 hours in Munin monitoring
• Each accepted-but-stuck connection contributes one FD
• At ~1024 FDs the daemon begins logging "Too many open files"
• apcaccess and other NIS clients stop receiving responses
════════════════════════════════════════════════════════════════
ROOT CAUSE — DETAILED
════════════════════════════════════════════════════════════════
─── Bug 1: src/lib/apclibnis.c, write_nbytes() ──────────────────
read_nbytes() guards recv() with a 15-second select() timeout. write_nbytes()
calls send() directly in a bare loop — no select(), no timeout:
Before (stock):
while (nleft > 0) {
nwritten = send(fd, ptr, nleft, 0); /* blocks indefinitely */
switch (nwritten) { ... }
}
When a peer's TCP receive window reaches zero (receive buffer full, not
reading), send() on a blocking socket blocks until the window reopens — which
never happens for a broken connection. Linux's default TCP retry period
(tcp_retries2 = 15 retransmits, exponential backoff) means ~15 minutes before
the kernel abandons the connection.
─── Bug 2: src/apcnis.c, status_close() ────────────────────────
The NIS status mutex is acquired in status_open() and released inside
status_close() — AFTER all network sends complete. status_close() sends the
entire status response (header + 40 lines + terminator) as individual
net_send() calls, each of which calls write_nbytes(). In the original code:
static int status_close(UPSINFO *ups, int nsockfd) {
int i = strlen(largebuf);
asnprintf(buf, ...); /* mutex still held */
net_send(nsockfd, buf, ...); /* net I/O under mutex */
for (...) {
net_send(nsockfd, sptr, ...); /* 40+ net I/O calls under mutex */
}
net_send(nsockfd, NULL, 0); /* terminator, under mutex */
V(mutex); /* released only after all I/O completes */
}
One thread blocked in write_nbytes() prevents ALL other connection threads
from acquiring the mutex. Every subsequent client connection spawns a thread
that queues on P(mutex), holding its accepted socket FD. The FDs accumulate.
═══════════════════════════════════════════════════════════════
REPRODUCTION
═══════════════════════════════════════════════════════════════
The test requires constraining the system TCP send buffer so the server's
~1 KB status response fills it (default auto-tune allows the kernel to
buffer the entire response and return from send() immediately):
Step 1 — Constrain TCP send buffer (temporarily):
sudo sysctl -w net.ipv4.tcp_wmem='4096 4096 4096'
sudo systemctl restart apcupsd
Step 2 — Run the test script (see below) in one terminal:
python3 nis_hang_test.py 127.0.0.1 4 10 120
Step 3 — In another terminal, sample the apcupsd thread count every 2s:
APCPID=$(systemctl show -p MainPID --value apcupsd)
for t in $(seq 2 2 30); do
sleep 2
echo "${t}s: Threads=$(ls /proc/$APCPID/task | wc -l) FDs=$(ls
/proc/$APCPID/fd | wc -l)"
done
Stock binary result — threads stuck indefinitely:
2s: Threads=7 FDs=12 (baseline is Threads=3 FDs=7)
4s: Threads=7 FDs=12
...
28s: Threads=7 FDs=12 (still stuck; would remain until ~15 min TCP retry)
30s: Threads=7 FDs=12
Patched binary result — threads recover after 15-second write timeout:
2s: Threads=7 FDs=12
4s: Threads=7 FDs=12
...
28s: Threads=7 FDs=12
30s: Threads=3 FDs=8 ← write timeout fired, mutex released, FDs freed
Step 4 — Restore TCP send buffer:
sudo sysctl -w net.ipv4.tcp_wmem='4096 16384 4194304'
sudo systemctl restart apcupsd
── Test script (nis_hang_test.py) ──────────────────────────────
#!/usr/bin/env python3
"""
Open N connections to apcupsd port 3551. Each connection queues N_REQUESTS
'status' commands but never reads responses. SO_RCVBUF is forced to minimum
so the server's send buffer fills after ~3 responses, causing send() to block.
With stock code: server threads hang indefinitely, holding the mutex.
With patched code: threads time out after 15s and exit cleanly.
Usage: nis_hang_test.py [HOST] [N_CONNS] [N_REQUESTS] [WAIT_SECS]
"""
import socket, struct, time, sys
HOST = sys.argv[1] if len(sys.argv) > 1 else '127.0.0.1'
N_CONNS = int(sys.argv[2]) if len(sys.argv) > 2 else 4
N_REQ = int(sys.argv[3]) if len(sys.argv) > 3 else 10
WAIT = int(sys.argv[4]) if len(sys.argv) > 4 else 120
PORT = 3551
def make_request(cmd):
data = cmd.encode()
return struct.pack('!H', len(data)) + data
sockets = []
for i in range(N_CONNS):
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 1)
s.connect((HOST, PORT))
s.sendall(b''.join(make_request('status') for _ in range(N_REQ)))
sockets.append(s)
print(f" conn {i+1}: SO_RCVBUF={s.getsockopt(socket.SOL_SOCKET,
socket.SO_RCVBUF)}, queued {N_REQ} requests")
except Exception as e:
print(f" conn {i+1}: failed: {e}")
print(f"\n{len(sockets)} connections open. Server threads should now be
blocked.")
time.sleep(WAIT)
for s in sockets: s.close()
════════════════════════════════════════════════════════════════
FIX
════════════════════════════════════════════════════════════════
Two changes, combined in the patch below:
Fix 1 — Add a 15-second select() timeout to write_nbytes(), matching the
existing timeout in read_nbytes(). A blocked send() now times out,
returns -ETIMEDOUT, and the connection is closed cleanly.
Fix 2 — In status_close(), snapshot largebuf into a local stack buffer and
release the mutex BEFORE any network I/O. Other threads can now
serve their own connections while one thread is sending. A
secondary benefit: largebuf_len is tracked incrementally in
status_write() instead of recomputing strlen(largebuf) each time.
── Patch ──────────────────────────────────────────────────────
--- src/lib/apclibnis.c.orig
+++ src/lib/apclibnis.c
@@ -99,23 +99,46 @@
/*
* Write nbytes to the network.
* It may require several writes.
+ * Uses select() with a 15-second timeout to match read_nbytes(). Without a
+ * send timeout, a connection whose peer stops consuming data (broken TCP in
+ * ESTABLISHED state, no RST/FIN) blocks send() for the kernel's TCP retry
+ * period (~15 min), holding the NIS status mutex and causing FD exhaustion.
*/
static int write_nbytes(sock_t fd, const char *ptr, int nbytes)
{
int nleft, nwritten;
+ struct timeval timeout;
+ int rc;
+ fd_set fds;
nleft = nbytes;
while (nleft > 0) {
- nwritten = send(fd, ptr, nleft, 0);
+ do {
+ /* Wait up to 15 seconds for the socket to accept data */
+ timeout.tv_sec = 15;
+ timeout.tv_usec = 0;
- switch (nwritten) {
- case -1:
- if (errno == EINTR || errno == EAGAIN)
- continue;
- return -errno; /* error */
- case 0:
- return nbytes - nleft; /* EOF */
- }
+ FD_ZERO(&fds);
+ FD_SET(fd, &fds);
+
+ rc = select(fd + 1, NULL, &fds, NULL, &timeout);
+
+ switch (rc) {
+ case -1:
+ if (errno == EINTR || errno == EAGAIN)
+ continue;
+ return -errno; /* error */
+ case 0:
+ return -ETIMEDOUT; /* timeout */
+ }
+
+ nwritten = send(fd, ptr, nleft, 0);
+ } while (nwritten == -1 && (errno == EINTR || errno == EAGAIN));
+
+ if (nwritten == 0)
+ return nbytes - nleft; /* EOF */
+ if (nwritten < 0)
+ return -errno; /* error */
nleft -= nwritten;
ptr += nwritten;
--- src/apcnis.c.orig
+++ src/apcnis.c
@@ -31,6 +31,7 @@
static char largebuf[4096];
static int stat_recs;
+static int largebuf_len;
struct s_arg {
UPSINFO *ups;
@@ -45,14 +46,18 @@
{
P(mutex);
largebuf[0] = 0;
+ largebuf_len = 0;
stat_recs = 0;
}
#define STAT_REV 1
/*
- * Send the status lines across the network one line
- * at a time (to prevent sending too large a buffer).
+ * Send the status lines across the network one line at a time.
+ *
+ * The shared largebuf is copied into a local buffer before releasing the
+ * mutex so that network I/O (which may block under a write_nbytes timeout)
+ * does not hold the mutex and stall all other connection threads.
*
* Returns -1 on error or EOF
* 0 OK
@@ -60,19 +65,21 @@
static int status_close(UPSINFO *ups, int nsockfd)
{
int i;
- char buf[MAXSTRING];
+ char hdr[MAXSTRING];
+ char sendbuf[sizeof(largebuf)];
char *sptr, *eptr;
- i = strlen(largebuf);
- asnprintf(buf, sizeof(buf), "APC : %03d,%03d,%04d\n",
+ /* Snapshot shared state and release the lock before any network I/O. */
+ i = largebuf_len;
+ asnprintf(hdr, sizeof(hdr), "APC : %03d,%03d,%04d\n",
STAT_REV, stat_recs, i);
+ memcpy(sendbuf, largebuf, i + 1);
+ V(mutex);
- if (net_send(nsockfd, buf, strlen(buf)) <= 0) {
- V(mutex);
+ if (net_send(nsockfd, hdr, strlen(hdr)) <= 0)
return -1;
- }
- sptr = eptr = largebuf;
+ sptr = eptr = sendbuf;
for (; i > 0; i--) {
if (*eptr == '\n') {
eptr++;
@@ -84,12 +91,9 @@
}
}
- if (net_send(nsockfd, NULL, 0) < 0) {
- V(mutex);
+ if (net_send(nsockfd, NULL, 0) < 0)
return -1;
- }
- V(mutex);
return 0;
}
@@ -109,8 +113,9 @@
avsnprintf(buf, sizeof(buf), fmt, ap);
va_end(ap);
- if ((i = (strlen(largebuf) + strlen(buf))) < (int)(sizeof(largebuf) - 1)) {
+ if ((i = (largebuf_len + strlen(buf))) < (int)(sizeof(largebuf) - 1)) {
strlcat(largebuf, buf, sizeof(largebuf));
+ largebuf_len = i;
stat_recs++;
} else {
log_event(ups, LOG_ERR,
════════════════════════════════════════════════════════════════
NOTES
════════════════════════════════════════════════════════════════
• Both bugs are in the upstream 3.14.14 source (released May 2016) and are
present in all versions that reporter has inspected.
• The TCP send buffer constraint (tcp_wmem sysctl) is required for the
synthetic test because the default auto-tuned buffer (~100 KB) is large
enough to hold the entire status response in flight without blocking send().
The production failure condition — a long-lived ESTABLISHED connection whose
peer stops ACKing (NAT expiry, firewall change, network partition) — fills
the send buffer naturally over time as unACKed data accumulates.
• The asymmetry between read_nbytes() and write_nbytes() suggests the send
timeout was simply overlooked when the select()-based read guard was added.
• The events-file path (output_events()) sends data via the same net_send()
path but does NOT hold the mutex, so it is not subject to Bug 2. However,
it is subject to Bug 1 (write_nbytes timeout), and a slow/broken client
there can also stall a thread, consuming a thread and its FD indefinitely.
• The 15-second write timeout is intentionally generous: a ~1 KB status
response should be deliverable in well under a second on any reasonable
path. A timeout at this threshold is unlikely to affect healthy clients
under normal conditions, though deployments over heavily congested or
high-latency links may wish to adjust the value.
• Known pre-existing edge case (not introduced by this patch): if a send
failure occurs partway through the status line loop in status_close(),
the existing code breaks out of the loop and still attempts to send the
zero-length terminator record. If the terminator send succeeds, the
function returns success despite having sent a truncated response; the
client receives fewer lines than the APC header advertises but sees a
well-formed terminator and may display incomplete data without any
error indication. In the failure scenario this patch addresses (send
buffer full, peer not reading) the terminator send will also fail, so
clients receive nothing rather than partial data. Fixing the partial-
response path cleanly would require tracking loop completion separately
and is left as a follow-up.
• This analysis, test script, and patch were developed with the assistance
of Claude Code (Anthropic), model claude-sonnet-4-6, on 2026-05-15.
ProblemType: Bug
DistroRelease: Ubuntu 26.04
Package: apcupsd 3.14.14-7 [modified: usr/sbin/apcupsd]
ProcVersionSignature: Ubuntu 7.0.0-15.15-generic 7.0.0
Uname: Linux 7.0.0-15-generic x86_64
ApportVersion: 2.34.0-0ubuntu2
Architecture: amd64
CasperMD5CheckResult: unknown
Date: Fri May 15 10:23:06 2026
SourcePackage: apcupsd
UpgradeStatus: Upgraded to resolute on 2026-05-03 (12 days ago)
modified.conffile..etc.apcupsd.apcupsd.conf: [modified]
modified.conffile..etc.default.apcupsd: [modified]
mtime.conffile..etc.apcupsd.apcupsd.conf: 2022-05-04T16:58:46.163139
mtime.conffile..etc.default.apcupsd: 2021-02-16T15:22:42.669805
** Affects: apcupsd (Ubuntu)
Importance: Undecided
Status: New
** Tags: amd64 apport-bug resolute
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2152692
Title:
apcupsd NIS server deadlock: write_nbytes() has no send timeout, mutex
held across network I/O → FD exhaustion under broken TCP
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apcupsd/+bug/2152692/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
