> On Sep 21, 2026, at 15:12, Bharath Rupireddy
> <[email protected]> wrote:
>
> Hi,
>
> A failed WAL read leaks the open WAL segment file descriptor until the
> backend exits. Once the fd limit is reached, all later queries in that
> backend fail with "Too many open files". With a connection pooler the
> leaks add up across clients. There is no warning or log message for
> this. Reproducers are at [1] and [2].
>
> It can happen with logical decoding that cancelled decode calls
> exhaust the fds, and depending on which open fails first, the backend
> dies with a FATAL or the instance restarts with a PANIC. [2] shows a
> PANIC.
>
> It can also happen that the leaked fd is on a WAL segment that has
> since been removed. The file is unlinked but its space is not freed
> until the backend goes away, so pg_wal still looks small while free
> space on the disk keeps shrinking. On a small WAL volume I could fill
> the disk this way, and the server then PANICed while writing a new WAL
> file with no space left, which took down the database instance.
>
> I noticed this while looking at the error paths in pg_walinspect for
> the reported issue [3]. I then used Claude Code to check for the same
> issue elsewhere, and it helped me find the other affected paths.
>
> The WAL read paths open the segment file as a plain kernel fd. It is
> not a virtual fd and not a transient file, and no resource owner owns
> it, so nothing in the backend knows it is open. On an error, the
> memory holding the WAL reader is freed with its context, but the
> segment file it had open stays open. This leak seems to exist in
> pg_walinspect functions, logical decoding functions, 2PC WAL read
> code, and the WAL summarizer. All of these except the WAL summarizer
> are reachable from SQL in simple ways. The walsender does not have
> this issue because it closes the file in its own error cleanup. The
> startup process is fine too, its reader does not keep a segment open
> this way, and its read errors are FATAL anyway.
>
> This is the same leak that commit 91c40548d5 fixed for 2PC by closing
> the fd in XLogReaderFree(), but that close only runs on the normal
> path, not when an error is thrown before it.
>
> I think the fix is to register a memory context reset callback on the
> context the reader is allocated in. If that context is reset or
> deleted while a segment file is still open, the callback closes it.
> Doing this in XLogReaderAllocate() covers every caller, present and
> future, instead of adding an error handler to each one. On HEAD,
> XLogReaderFree() unregisters the callback and the reader keeps a
> pointer to it in a new field. On PG18 and older the reader struct
> cannot grow because of ABI, and there is no
> MemoryContextUnregisterResetCallback(), so there the callback stays
> registered and its state lives in a small list local to the WAL
> reader. I attached a nocfbot prefixed patch for the back branches
> doing that.
>
> There are alternative approaches that have some issues. Wrapping each
> caller in its own error handler works but is easy to miss in future
> callers. Opening the segment as a transient file, so that it gets
> closed at transaction abort, adds a rule that a reader cannot outlive
> a transaction, which does not work for the logical walsender since it
> reads WAL outside any transaction. Tracking the fd with a resource
> owner also fixes all callers in one place, but the extensible resource
> owner API only exists in PG17 and later, so it cannot be used in the
> older branches.
>
> Please find the attached patches (v1 for HEAD and PG19, nocfbot for
> PG18). If the approach looks good, I will prepare patches for all the
> remaining back branches using the PG18 approach.
>
> Thoughts?
>
> [1]
> $ ulimit -n 200
>
> CREATE TABLE t1 (a int);
> INSERT INTO t1 VALUES (1);
>
> -- Each iteration errors at end of WAL and leaks one fd
> DO $$
> BEGIN
> FOR i IN 1..200 LOOP
> BEGIN
> PERFORM * FROM
> pg_get_wal_records_info(pg_current_wal_flush_lsn(),
> 'FFFFFFFF/FFFFFFFF');
> EXCEPTION WHEN OTHERS THEN
> NULL;
> END;
> END LOOP;
> END $$;
>
> -- Session can no longer open any file
> postgres=# SELECT count(*) FROM t1;
> ERROR: could not open file "base/5/2691": Too many open files
> LINE 1: SELECT count(*) FROM t1;
> ^
> postgres=# CREATE TABLE t2 (a int);
> ERROR: could not create file "base/5/16398": Too many open files
>
> -- Shows the fd table full of pg_wal segments
> $ lsof -w -p <backend pid> | grep -c pg_wal
> 193
>
> [2]
> $ ulimit -n 200
>
> SELECT pg_create_logical_replication_slot('s', 'test_decoding');
> CREATE TABLE foo(a int, b text);
> INSERT INTO foo SELECT i, repeat('x', 200) FROM generate_series(1, 300000) i;
>
> -- Each cancelled decode errors mid-read and leaks one fd
> SET statement_timeout = '10ms';
> SELECT 'SELECT * FROM pg_logical_slot_peek_changes(''s'', NULL, NULL);'
> FROM generate_series(1, 200) \gexec
>
> ERROR: canceling statement due to statement timeout
> ERROR: canceling statement due to statement timeout
> ..
> ..
> ERROR: canceling statement due to statement timeout
> ERROR: canceling statement due to statement timeout
> PANIC: could not open file "pg_logical/snapshots/0-18692A8.snap": Too
> many open files
> server closed the connection unexpectedly
> This probably means the server terminated abnormally
> before or while processing the request.
> The connection to the server was lost. Attempting reset: Failed.
>
> [3] https://postgr.es/m/[email protected]
>
> --
> Bharath Rupireddy
> Amazon Web Services: https://aws.amazon.com
> <v1-0001-Fix-WAL-segment-file-descriptor-leak-on-WAL-read-.patch><nocfbot-v1-0001-PG18-Fix-WAL-segment-file-descriptor-leak.patch>
I have objection on releasing the FD by a callback. A few comments on v1:
1
```
+ * Reset callbacks run before the context's memory is freed, so the reader is
+ * still valid here. segment_close must not throw an error.
+ */
+static void
+xlogreader_reset_callback(void *arg)
```
This patch seems to introduce a new requirement for segment_close: it must not
throw an error when invoked from the reset callback.
Should this requirement also be documented in the header comment for
segment_close itself, so that it becomes part of the callback contract?
2
```
+ /*
+ * Reset callback on the memory context holding this reader, which
closes
+ * the open WAL segment file if that context goes away before
+ * XLogReaderFree() is reached. Backend only, NULL in frontend code.
+ */
+ struct MemoryContextCallback *reset_cb;
```
This field is only used in backend code. Should it also be wrapped in #ifndef
FRONTEND?
3
```
+ /*
+ * The WAL segment file is opened with BasicOpenFile(), so nothing but
+ * XLogReaderFree() ever closes it. An error thrown while reading WAL
does
+ * not get that far, and the descriptor would then be leaked for the
life
+ * of the process, so close it on a reset of the context we are
allocated
+ * in as well.
+ */
+ state->reset_cb = palloc_extended(sizeof(MemoryContextCallback),
+
MCXT_ALLOC_NO_OOM | MCXT_ALLOC_ZERO);
```
I think MCXT_ALLOC_NO_OOM is important here, because at this point the reset
callback has not been registered yet. If this allocation itself threw an ERROR,
there would still be no callback available to clean up the reader's resources.
Would it be worth mentioning this in the comment, so the reason for using
MCXT_ALLOC_NO_OOM here is explicit?
Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/
