Hi,
While working on [1], I realized that the core dump output of OpenBSD
has very little information. We had an off-list chat with Thomas and
found that this happens because OpenBSD core dump files do not include
the executable information; so we need to run `lldb` with the
executable path. Attached patch manually finds executables in the
build directory and calls `lldb` with these executables' paths'.
Here is an example:
Master [2]:
```
[11:36:51.769] (lldb) target create --core "/var/crash/postgres.core"
[11:36:51.797] Core file '/var/crash/postgres.core' (x86_64) was loaded.
[11:36:51.797] (lldb) thread backtrace all
[11:36:51.798] * thread #1, stop reason = signal SIGABRT
[11:36:51.798] * frame #0: 0x0000029d071e14bb
[11:36:51.798] frame #1: 0x550327f2b69964ee
[11:36:51.798] (lldb) quit
```
Patched [3]:
```
[11:22:11.087] (lldb) target create
"/home/postgres/postgres/build/tmp_install/usr/local/pgsql/bin/postgres"
--core "/var/crash/postgres.core"
[11:22:13.104] Core file '/var/crash/postgres.core' (x86_64) was loaded.
[11:22:13.104] (lldb) thread backtrace all
[11:22:13.105] warning: This version of LLDB has no plugin for the
language "assembler". Inspection of frame variables will be limited.
[11:22:13.213] * thread #1, stop reason = signal SIGABRT
[11:22:13.213] * frame #0: 0x00000b12e4dfe27b
libc.so.100.3`_thread_sys_thrkill at -:2
[11:22:13.213] frame #1: 0x00000b12e4db010b
libc.so.100.3`_libc_abort at abort.c:51:8
[11:22:13.213] frame #2: 0x00000b126d3fd39d
pg_prewarm.so`pg_prewarm(fcinfo=<unavailable>) at pg_prewarm.c:80:2
[11:22:13.213] frame #3: 0x00000b0ff584add5
postgres`ExecInterpExpr(state=0x00000b12527e4790,
econtext=0x00000b12527e4438, isnull=0x0000000000000000) at
execExprInterp.c:926:8
[11:22:13.213] frame #4: 0x00000b0ff5896a47 postgres`ExecResult
[inlined] ExecEvalExprNoReturn(state=0x00000b12527e4790,
econtext=0x00000b12527e4438) at executor.h:420:13
[11:22:13.213] frame #5: 0x00000b0ff5896a3b postgres`ExecResult
[inlined] ExecEvalExprNoReturnSwitchContext(state=0x00000b12527e4790,
econtext=0x00000b12527e4438) at executor.h:461:2
[11:22:13.213] frame #6: 0x00000b0ff5896a2a postgres`ExecResult
[inlined] ExecProject(projInfo=0x00000b12527e4788) at executor.h:493:2
[11:22:13.213] frame #7: 0x00000b0ff5896a11
postgres`ExecResult(pstate=<unavailable>) at nodeResult.c:135:10
[11:22:13.213] frame #8: 0x00000b0ff5855eda
postgres`standard_ExecutorRun [inlined]
ExecProcNode(node=0x00000b12527e4328) at executor.h:316:9
[11:22:13.213] frame #9: 0x00000b0ff5855ec5
postgres`standard_ExecutorRun [inlined]
ExecutePlan(queryDesc=0x00000b124f332500, operation=CMD_SELECT,
sendTuples=<unavailable>, numberTuples=0, direction=<unavailable>,
dest=0x00000b12e5cae7a0) at execMain.c:1697:10
[11:22:13.213] frame #10: 0x00000b0ff5855e4a
postgres`standard_ExecutorRun(queryDesc=0x00000b124f332500,
direction=<unavailable>, count=0) at execMain.c:366:3
[11:22:13.213] frame #11: 0x00000b0ff5a7623b
postgres`PortalRunSelect(portal=0x00000b128d07c100,
forward=<unavailable>, count=0, dest=<unavailable>) at pquery.c:921:4
[11:22:13.213] frame #12: 0x00000b0ff5a75d9d
postgres`PortalRun(portal=0x00000b128d07c100,
count=9223372036854775807, isTopLevel=<unavailable>,
dest=0x00000b12e5cae7a0, altdest=0x00000b12e5cae7a0,
qc=0x0000759da39141b0) at pquery.c:765:18
[11:22:13.213] frame #13: 0x00000b0ff5a74ad9
postgres`exec_simple_query(query_string="SELECT pg_prewarm('test',
'read');") at postgres.c:1279:10
[11:22:13.213] frame #14: 0x00000b0ff5a72332
postgres`PostgresMain(dbname=<unavailable>, username=<unavailable>) at
postgres.c:0
[11:22:13.213] frame #15: 0x00000b0ff5a6bfad
postgres`BackendMain(startup_data=<unavailable>,
startup_data_len=<unavailable>) at backend_startup.c:124:2
[11:22:13.213] frame #16: 0x00000b0ff59a35a6
postgres`postmaster_child_launch(child_type=B_BACKEND, child_slot=1,
startup_data=0x0000759da3915030, startup_data_len=24,
client_sock=0x0000759da3914f20) at launch_backend.c:268:3
[11:22:13.213] frame #17: 0x00000b0ff59a8584 postgres`ServerLoop
[inlined] BackendStartup(client_sock=<unavailable>) at
postmaster.c:3598:8
[11:22:13.213] frame #18: 0x00000b0ff59a8543 postgres`ServerLoop
at postmaster.c:1713:6
[11:22:13.213] frame #19: 0x00000b0ff59a60e8
postgres`PostmasterMain(argc=<unavailable>, argv=<unavailable>) at
postmaster.c:1403:11
[11:22:13.213] frame #20: 0x00000b0ff58c6915 postgres`main(argc=4,
argv=0x0000759da39155c8) at main.c:231:4
[11:22:13.213] frame #21: 0x00000b0ff565728b postgres`__start + 299
```
Any feedback would be appreciated.
[1]
https://postgr.es/m/CAN55FZ12R-LG=wZZ8yLY7+tmVbR=9y_o5uvedhu0nmyt+ai...@mail.gmail.com
[2] https://cirrus-ci.com/task/6177664672727040
[3] https://cirrus-ci.com/task/5487875443130368
--
Regards,
Nazir Bilal Yavuz
Microsoft
From 01bf84431f3851cd579475ae947e6b78352485f0 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <[email protected]>
Date: Fri, 24 Oct 2025 13:29:30 +0300
Subject: [PATCH v1] ci: Improve OpenBSD core dump backtrace handling
Since OpenBSD core dumps do not embed executable paths, the script now
searches for the corresponding binary manually within the specified
directory before invoking LLDB.
---
.cirrus.tasks.yml | 3 ++-
src/tools/ci/cores_backtrace.sh | 27 +++++++++++++++++++++++----
2 files changed, 25 insertions(+), 5 deletions(-)
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index eca9d62fc22..6c6442eafef 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -322,6 +322,7 @@ task:
OS_NAME: openbsd
IMAGE_FAMILY: pg-ci-openbsd-postgres
PKGCONFIG_PATH: '/usr/lib/pkgconfig:/usr/local/lib/pkgconfig'
+ CORE_DUMP_EXECUTABLE_DIR: $CIRRUS_WORKING_DIR/build/tmp_install
MESON_FEATURES: >-
-Dbsd_auth=enabled
@@ -388,7 +389,7 @@ task:
# ${CORE_DUMP_DIR}, they may not obey this. So, move core files to the
# ${CORE_DUMP_DIR} directory.
find build/ -type f -name '*.core' -exec mv '{}' ${CORE_DUMP_DIR} \;
- src/tools/ci/cores_backtrace.sh ${OS_NAME} ${CORE_DUMP_DIR}
+ src/tools/ci/cores_backtrace.sh ${OS_NAME} ${CORE_DUMP_DIR} ${CORE_DUMP_EXECUTABLE_DIR}
# configure feature flags, shared between the task running the linux tests and
diff --git a/src/tools/ci/cores_backtrace.sh b/src/tools/ci/cores_backtrace.sh
index 54607415258..e334dc1fca7 100755
--- a/src/tools/ci/cores_backtrace.sh
+++ b/src/tools/ci/cores_backtrace.sh
@@ -1,12 +1,18 @@
#! /bin/sh
-if [ $# -ne 2 ]; then
+os=$1
+directory=$2
+executable_directory=$3
+
+if [ "$os" != 'openbsd' ] && [ $# -ne 2 ]; then
echo "cores_backtrace.sh <os> <directory>"
exit 1
fi
-os=$1
-directory=$2
+if [ "$os" = 'openbsd' ] && [ $# -ne 3 ]; then
+ echo "cores_backtrace.sh <os> <core_directory> <executable_directory>"
+ exit 1
+fi
case $os in
freebsd|linux|macos|netbsd|openbsd)
@@ -26,8 +32,21 @@ for corefile in $(find "$directory" -type f) ; do
echo -e '\n\n'
fi
- if [ "$os" = 'macos' ] || [ "$os" = 'openbsd' ]; then
+ if [ "$os" = 'macos' ]; then
lldb -c $corefile --batch -o 'thread backtrace all' -o 'quit'
+ elif [ "$os" = 'openbsd' ]; then
+ # OpenBSD's ELF format doesn't include executable information, so we
+ # search for the executable manually in <executable_directory>.
+ filename=$(basename "$corefile")
+ base=$(echo "$filename" | sed 's/\.core.*$//')
+ binary=$(find "$executable_directory" -type f -name "$base" 2>/dev/null | head -n 1)
+
+ if [ -z "$binary" ]; then
+ echo "${base} executable can not be found in ${executable_directory}"
+ continue
+ fi
+
+ lldb "$binary" -c "$corefile" --batch -o 'thread backtrace all' -o 'quit'
else
auxv=$(gdb --quiet --core ${corefile} --batch -ex 'info auxv' 2>/dev/null)
if [ $? -ne 0 ]; then
--
2.51.0
