I find that if I run the following test against a standard debug build
on HEAD, my local installation reliably segfaults:
$ meson test --setup running --suite test_rls_hooks-running
Attached is a "bt full" run from gdb against a core dump. The query
"EXPLAIN (costs off) SELECT * FROM rls_test_permissive;" runs when the
backend segfaults.
The top frame of the back trace is suggestive of a use-after-free:
#0 copyObjectImpl (from=0x7f7f7f7f7f7f7f7e) at copyfuncs.c:187
187 switch (nodeTag(from))
...
"git bisect" suggests that the problem began at commit 6ee30209,
"SQL/JSON: support the IS JSON predicate".
It's a bit surprising that the bug reproduces when I run a standard
test, and yet we appear to have a bug that's about 2 weeks old. There
may be something unusual about my system that will turn out to be
relevant -- though there is nothing particularly exotic about this
machine. My repro doesn't rely on concurrent execution, or timing, or
anything like that -- it's quite reliable.
--
Peter Geoghegan
2023-04-13 20:56:32.345 PDT [72281][postmaster] LOG: redirecting log output to
logging collector process
2023-04-13 20:56:32.345 PDT [72281][postmaster] HINT: Future log output will
appear in directory "log".
PID: 72308 (postgres)
UID: 1000 (pg)
GID: 1000 (pg)
Signal: 11 (SEGV)
Timestamp: Thu 2023-04-13 20:56:42 PDT (1s ago)
Command Line: $'postgres: pg regression_test_rls_hooks [local] EXPLAIN'
Executable: /mnt/nvme/postgresql/patch/install/bin/postgres
Control Group: /system.slice/ssh.service
Unit: ssh.service
Slice: system.slice
Boot ID: 73f38b1da37d4e2eba4b9c317530325b
Machine ID: 68e529f9438c4d718bc772fcb4e3753d
Hostname: horse
Storage:
/var/lib/systemd/coredump/core.postgres.1000.73f38b1da37d4e2eba4b9c317530325b.72308.1681444602000000.zst
(present)
Size on Disk: 1.6M
Message: Process 72308 (postgres) of user 1000 dumped core.
Stack trace of thread 72308:
#0 0x00005608fc99b3c0 copyObjectImpl (postgres + 0x5ad3c0)
#1 0x00005608fc99d6f0 _copyJoinExpr (postgres + 0x5af6f0)
#2 0x00005608fc99b68b copyObjectImpl (postgres + 0x5ad68b)
#3 0x00005608fc99dba2 _copyA_Expr (postgres + 0x5afba2)
#4 0x00005608fc99b6e6 copyObjectImpl (postgres + 0x5ad6e6)
#5 0x00007fdcc1533371 test_rls_hooks_permissive
(test_rls_hooks.so + 0x1371)
#6 0x00005608fc7cb32b get_policies_for_relation (postgres +
0x3dd32b)
#7 0x00005608fc7cb79f get_row_security_policies (postgres +
0x3dd79f)
#8 0x00005608fc7c4727 fireRIRrules (postgres + 0x3d6727)
#9 0x00005608fc7c78f3 QueryRewrite (postgres + 0x3d98f3)
#10 0x00005608fc5e9399 ExplainQuery (postgres + 0x1fb399)
#11 0x00005608fc8193fd standard_ProcessUtility (postgres +
0x42b3fd)
#12 0x00005608fc8198a2 ProcessUtility (postgres + 0x42b8a2)
#13 0x00005608fc816cc4 PortalRunUtility (postgres + 0x428cc4)
#14 0x00005608fc817112 FillPortalStore (postgres + 0x429112)
#15 0x00005608fc8174e7 PortalRun (postgres + 0x4294e7)
#16 0x00005608fc8133d2 exec_simple_query (postgres + 0x4253d2)
#17 0x00005608fc8153af PostgresMain (postgres + 0x4273af)
#18 0x00005608fc77d946 BackendRun (postgres + 0x38f946)
#19 0x00005608fc77fa4a BackendStartup (postgres + 0x391a4a)
#20 0x00005608fc77fbec ServerLoop (postgres + 0x391bec)
#21 0x00005608fc7811ee PostmasterMain (postgres + 0x3931ee)
#22 0x00005608fc6c405e main (postgres + 0x2d605e)
#23 0x00007fdcc084618a __libc_start_call_main (libc.so.6 +
0x2718a)
#24 0x00007fdcc0846245 __libc_start_main_impl (libc.so.6 +
0x27245)
#25 0x00005608fc4bfea1 _start (postgres + 0xd1ea1)
ELF object binary architecture: AMD x86-64
GNU gdb (Debian 13.1-2) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /mnt/nvme/postgresql/patch/install/bin/postgres...
warning: Can't open file /dev/zero (deleted) during file-backed mapping note
processing
warning: Can't open file /dev/shm/PostgreSQL.1471391328 during file-backed
mapping note processing
warning: Can't open file /dev/shm/PostgreSQL.1114565772 during file-backed
mapping note processing
warning: Can't open file /SYSV00000094 (deleted) during file-backed mapping
note processing
[New LWP 72308]
warning: Section `.reg-xstate/72308' in core file too small.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `postgres: pg regression_test_rls_hooks [local] EXPLAIN
'.
Program terminated with signal SIGSEGV, Segmentation fault.
warning: Section `.reg-xstate/72308' in core file too small.
#0 copyObjectImpl (from=0x7f7f7f7f7f7f7f7e) at copyfuncs.c:187
187 switch (nodeTag(from))
(gdb) bt full
#0 copyObjectImpl (from=0x7f7f7f7f7f7f7f7e) at copyfuncs.c:187 retval =
<optimized out>
__func__ = "copyObjectImpl"
#1 0x00005608fc99d6f0 in _copyJoinExpr (from=from@entry=0x5608fd53dfc0) at
copyfuncs.funcs.c:811
newnode = 0x5608fd539628
#2 0x00005608fc99b68b in copyObjectImpl (from=0x5608fd53dfc0) at
copyfuncs.switch.c:175
retval = <optimized out>
__func__ = "copyObjectImpl"
#3 0x00005608fc99dba2 in _copyA_Expr (from=from@entry=0x5608fd53e060) at
copyfuncs.funcs.c:947
newnode = 0x5608fd53e120
#4 0x00005608fc99b6e6 in copyObjectImpl (from=0x5608fd53e060) at
copyfuncs.switch.c:196
retval = <optimized out>
__func__ = "copyObjectImpl"
#5 0x00007fdcc1533371 in test_rls_hooks_permissive (cmdtype=<optimized out>,
relation=<optimized out>) at test_rls_hooks.c:90
policies = 0x0
policy = 0x5608fd545e68
role = 0
n = 0x5608fd53df70
e = <optimized out>
c = 0x5608fd53dfc0
qual_pstate = 0x5608fd5396b8
nsitem = <optimized out>
#6 0x00005608fc7cb32b in get_policies_for_relation
(relation=relation@entry=0x7fdcbe893278, cmd=cmd@entry=CMD_SELECT,
user_id=user_id@entry=16400,
permissive_policies=permissive_policies@entry=0x7ffd3fb43638,
restrictive_policies=restrictive_policies@entry=0x7ffd3fb43630) at
rowsecurity.c:605
hook_policies = <optimized out>
item = <optimized out>
__func__ = "get_policies_for_relation"
#7 0x00005608fc7cb79f in get_row_security_policies
(root=root@entry=0x5608fd465008, rte=rte@entry=0x5608fd465118,
rt_index=rt_index@entry=1,
securityQuals=securityQuals@entry=0x7ffd3fb436c0,
withCheckOptions=withCheckOptions@entry=0x7ffd3fb436c8,
hasRowSecurity=hasRowSecurity@entry=0x7ffd3fb436bd,
hasSubLinks=0x7ffd3fb436be) at rowsecurity.c:229
user_id = 16400
rls_status = <optimized out>
rel = 0x7fdcbe893278
commandType = <optimized out>
permissive_policies = 0x0
restrictive_policies = 0x0
perminfo = 0x5608fd465248
#8 0x00005608fc7c4727 in fireRIRrules (parsetree=0x5608fd465008,
activeRIRs=activeRIRs@entry=0x0) at rewriteHandler.c:2186
rte = 0x5608fd465118
rel = 0x7fdcbe893278
withCheckOptions = 0x0
hasRowSecurity = false
hasSubLinks = false
securityQuals = 0x0
lc__state = <optimized out>
origResultRelation = <optimized out>
rt_index = 1
lc = <optimized out>
__func__ = "fireRIRrules"
#9 0x00005608fc7c78f3 in QueryRewrite (parsetree=0x5608fd465008) at
rewriteHandler.c:4231
query = <optimized out>
l__state = <optimized out>
input_query_id = 0
querylist = 0x5608fd541bd0
results = 0x0
l = <optimized out>
origCmdType = <optimized out>
foundOriginalQuery = <optimized out>
lastInstead = <optimized out>
#10 0x00005608fc5e9399 in ExplainQuery (pstate=pstate@entry=0x5608fd545d58,
stmt=stmt@entry=0x5608fd464e48, params=params@entry=0x0,
dest=dest@entry=0x5608fd541a58) at explain.c:269
es = 0x5608fd541ae8
tstate = <optimized out>
jstate = <optimized out>
query = 0x5608fd465008
rewritten = <optimized out>
lc = <optimized out>
timing_set = false
summary_set = false
__func__ = "ExplainQuery"
#11 0x00005608fc8193fd in standard_ProcessUtility (pstmt=0x5608fd464ef8,
queryString=0x5608fd464038 "EXPLAIN (costs off) SELECT * FROM
rls_test_permissive;",
readOnlyTree=<optimized out>, context=PROCESS_UTILITY_TOPLEVEL, params=0x0,
queryEnv=0x0, dest=0x5608fd541a58, qc=0x7ffd3fb438d0) at utility.c:870
parsetree = 0x5608fd464e48
isTopLevel = true
isAtomicContext = false
pstate = 0x5608fd545d58
readonly_flags = <optimized out>
__func__ = "standard_ProcessUtility"
#12 0x00005608fc8198a2 in ProcessUtility (pstmt=pstmt@entry=0x5608fd464ef8,
queryString=<optimized out>, readOnlyTree=<optimized out>,
context=context@entry=PROCESS_UTILITY_TOPLEVEL,
params=<optimized out>, queryEnv=<optimized out>, dest=0x5608fd541a58,
qc=0x7ffd3fb438d0) at utility.c:530
No locals.
#13 0x00005608fc816cc4 in PortalRunUtility (portal=portal@entry=0x5608fd4e5e08,
pstmt=0x5608fd464ef8, isTopLevel=isTopLevel@entry=true,
setHoldSnapshot=setHoldSnapshot@entry=true,
dest=dest@entry=0x5608fd541a58, qc=qc@entry=0x7ffd3fb438d0) at pquery.c:1158
No locals.
#14 0x00005608fc817112 in FillPortalStore (portal=portal@entry=0x5608fd4e5e08,
isTopLevel=isTopLevel@entry=true) at pquery.c:1031
treceiver = 0x5608fd541a58
qc = {
commandTag = CMDTAG_UNKNOWN,
nprocessed = 0
}
__func__ = "FillPortalStore"
#15 0x00005608fc8174e7 in PortalRun (portal=portal@entry=0x5608fd4e5e08,
count=count@entry=9223372036854775807, isTopLevel=isTopLevel@entry=true,
run_once=run_once@entry=true,
dest=dest@entry=0x5608fd4655b0, altdest=altdest@entry=0x5608fd4655b0,
qc=0x7ffd3fb43ad0) at pquery.c:763
_save_exception_stack = 0x7ffd3fb43b70
_save_context_stack = 0x0
_local_sigjmp_buf = {
[0] = {
__jmpbuf = {
[0] = 94596609197488,
[1] = 5120444486924160828,
[2] = 94596608970360,
[3] = 94596608970408,
[4] = 94596609498632,
[5] = 94596608972208,
[6] = 5120444486833983292,
[7] = 1505608874091925308
},
__mask_was_saved = 0,
__saved_mask = {
__val = {
[0] = 8,
[1] = 18248645120,
[2] = 112,
[3] = 94596599048070,
[4] = 112,
[5] = 94596608970360,
[6] = 94596608966432,
[7] = 94596608970432,
[8] = 153,
[9] = 140725672229440,
[10] = 94596597482311,
[11] = 94596609498632,
[12] = 2,
[13] = 94596608970360,
[14] = 94596608970408,
[15] = 140725672229472
}
}
}
}
_do_rethrow = <optimized out>
result = <optimized out>
nprocessed = <optimized out>
saveTopTransactionResourceOwner = 0x5608fd4a5628
saveTopTransactionContext = 0x5608fd49c5b0
saveActivePortal = 0x0
saveResourceOwner = 0x5608fd4a5628
savePortalContext = 0x0
saveMemoryContext = 0x5608fd49c5b0
__func__ = "PortalRun"
#16 0x00005608fc8133d2 in exec_simple_query
(query_string=query_string@entry=0x5608fd464038 "EXPLAIN (costs off) SELECT *
FROM rls_test_permissive;") at postgres.c:1240
cmdtaglen = 7
snapshot_set = <optimized out>
per_parsetree_context = 0x0
plantree_list = 0x5608fd558308
parsetree = 0x5608fd464e78
commandTag = <optimized out>
qc = {
commandTag = CMDTAG_UNKNOWN,
nprocessed = 0
}
querytree_list = <optimized out>
portal = 0x5608fd4e5e08
receiver = 0x5608fd4655b0
format = 0
cmdtagname = <optimized out>
parsetree_item__state = <optimized out>
dest = DestRemote
oldcontext = 0x5608fd49c5b0
parsetree_list = 0x5608fd464ea8
parsetree_item = 0x5608fd464ec0
save_log_statement_stats = false
was_logged = false
use_implicit_block = false
msec_str =
"7\000\000\000\000\000\000\000@<\264?\375\177\000\000\000;\264?\375\177\000\000\021\220\225\374\bV\000"
__func__ = "exec_simple_query"
#17 0x00005608fc8153af in PostgresMain (dbname=<optimized out>,
username=<optimized out>) at postgres.c:4572
query_string = 0x5608fd464038 "EXPLAIN (costs off) SELECT * FROM
rls_test_permissive;"
firstchar = <optimized out>
input_message = {
data = 0x5608fd464038 "EXPLAIN (costs off) SELECT * FROM
rls_test_permissive;",
len = 55,
maxlen = 1024,
cursor = 55
}
local_sigjmp_buf = {
[0] = {
__jmpbuf = {
[0] = 94596601295544,
[1] = 5120444486716542780,
[2] = 94596601295552,
[3] = 94596609160848,
[4] = 1681444592,
[5] = 94596609160848,
[6] = 5120444486894800700,
[7] = 1505608874116173628
},
__mask_was_saved = 1,
__saved_mask = {
__val = {
[0] = 4194304,
[1] = 17088,
[2] = 140725672229936,
[3] = 17032,
[4] = 94596597446733,
[5] = 2,
[6] = 8192,
[7] = 44631117552,
[8] = 17024,
[9] = 140725672229936,
[10] = 94596608676736,
[11] = 17024,
[12] = 94596601289704,
[13] = 1681444592,
[14] = 94596609160848,
[15] = 140725672229968
}
}
}
}
send_ready_for_query = false
idle_in_transaction_timeout_enabled = false
idle_session_timeout_enabled = false
__func__ = "PostgresMain"
#18 0x00005608fc77d946 in BackendRun (port=port@entry=0x5608fd493690) at
postmaster.c:4461
No locals.
#19 0x00005608fc77fa4a in BackendStartup (port=port@entry=0x5608fd493690) at
postmaster.c:4189
bn = 0x5608fd4948c0
pid = 0
__func__ = "BackendStartup"
#20 0x00005608fc77fbec in ServerLoop () at postmaster.c:1779
port = 0x5608fd493690
i = 0
now = <optimized out>
last_lockfile_recheck_time = 1681444592
last_touch_time = 1681444592
events = {
[0] = {
pos = 3,
events = 2,
fd = 9,
user_data = 0x0
},
[1] = {
pos = -1065190840,
events = 32732,
fd = 1068776884,
user_data = 0x7fdcc183e1ba <do_lookup_x+954 at dl-lookup.c:420>
},
[2] = {
pos = 2500,
events = 0,
fd = -1064604035,
user_data = 0x7fdcc1546f60
},
[3] = {
pos = -45496912,
events = 22024,
fd = 2050,
user_data = 0x7ffd3fb43dc0
},
[4] = {
pos = -1063328288,
events = 32732,
fd = -1063314336,
user_data = 0x802
},
[5] = {
pos = -72,
events = 4294967295,
fd = 1,
user_data = 0x5608fd4657e8
},
[6] = {
pos = -45727688,
events = 22024,
fd = -1064601446,
user_data = 0x7ffd3fb43dd0
},
[7] = {
pos = -57102536,
events = 22024,
fd = 32768,
user_data = 0xa
},
[8] = {
pos = 19,
events = 0,
fd = -1064375387,
user_data = 0x7ffd3fb442e3
},
[9] = {
pos = 10,
events = 0,
fd = 1068776912,
user_data = 0x7fdcc08eec12 <opendir_tail+82 at opendir.c:64>
},
[10] = {
pos = 66304,
events = 0,
fd = 525520,
user_data = 0x100000002
},
[11] = {
pos = 16832,
events = 1000,
fd = 1000,
user_data = 0x96a2bfb414337100
},
[12] = {
pos = 6,
events = 0,
fd = -128,
user_data = 0x0
},
[13] = {
pos = -45496912,
events = 22024,
fd = 1,
user_data = 0x5608fd4657e8
},
[14] = {
pos = -45727688,
events = 22024,
fd = -1064600273,
user_data = 0x60293aa
},
[15] = {
pos = -1064374539,
events = 32732,
fd = 0,
user_data = 0xa
},
[16] = {
pos = 1068777104,
events = 32765,
fd = -1064375075,
user_data = 0x5608fd465f30
},
[17] = {
pos = -58855942,
events = 22024,
fd = 15,
user_data = 0x5608fd49c5b0
},
[18] = {
pos = 1068777136,
events = 32765,
fd = -58852310,
user_data = 0x1
},
[19] = {
pos = -45496821,
events = 22024,
fd = 1068778256,
user_data = 0x5608fc7e2230 <RemovePgTempFiles+294 at fd.c:3116>
},
[20] = {
pos = 1702060386,
events = 1936158767,
fd = 1952410737,
user_data = 0x13
},
[21] = {
pos = -1064604563,
events = 32732,
fd = -45727688,
user_data = 0x1820
},
[22] = {
pos = 255,
events = 0,
fd = 99,
user_data = 0x7fdcc08364a8
},
[23] = {
pos = -1063314336,
events = 32732,
fd = 255,
user_data = 0xffffffffffffffb8
},
[24] = {
pos = 1023,
events = 0,
fd = 0,
user_data = 0x5608fd4216c0
},
[25] = {
pos = -1064601446,
events = 32732,
fd = 15,
user_data = 0x5608fd4216c0
},
[26] = {
pos = 4096,
events = 0,
fd = 10,
user_data = 0x3ff
},
[27] = {
pos = -1064745083,
events = 32732,
fd = 66304,
user_data = 0x8b67c
},
[28] = {
pos = 1,
events = 0,
fd = 33152,
user_data = 0x3e8
},
[29] = {
pos = 0,
events = 0,
fd = 2640,
user_data = 0x5608fc6b9d46 <next_token+391 at hba.c:223>
},
[30] = {
pos = 8,
events = 0,
fd = 1068777567,
user_data = 0x7ffd3fb4405e
},
[31] = {
pos = 1068777568,
events = 32765,
fd = 1068777823,
user_data = 0x230000000000000f
},
[32] = {
pos = 116831652,
events = 0,
fd = 0,
user_data = 0x0
},
[33] = {
pos = 1068777976,
events = 32765,
fd = 72,
user_data = 0xf
},
[34] = {
pos = 1068777872,
events = 32765,
fd = -60053466,
user_data = 0x7ffd3fb441f8
},
[35] = {
pos = -1064689840,
events = 32732,
fd = -46000448,
user_data = 0x5608fd4216c0
},
[36] = {
pos = -1063331360,
events = 32732,
fd = -1064610437,
user_data = 0xffffffffff
},
[37] = {
pos = 0,
events = 0,
fd = 1937076736,
user_data = 0x5608fd4216c0
},
[38] = {
pos = -1063328288,
events = 32732,
fd = 338915584,
user_data = 0xc00
},
[39] = {
pos = -128,
events = 4294967295,
fd = 0,
user_data = 0x5608fd425ae0
},
[40] = {
pos = 1024,
events = 0,
fd = -45982784,
user_data = 0x0
},
[41] = {
pos = -1064600273,
events = 32732,
fd = -45983008,
user_data = 0x1
},
[42] = {
pos = 1024,
events = 0,
fd = -45553536,
user_data = 0x7ffd3fb44120
},
[43] = {
pos = -57243337,
events = 22024,
fd = -56148576,
user_data = 0x5608fd425ae0
},
[44] = {
pos = -45983008,
events = 22024,
fd = 1,
user_data = 0x400
},
[45] = {
pos = 0,
events = 0,
fd = 0,
user_data = 0x11a5d
},
[46] = {
pos = -128,
events = 4294967295,
fd = 0,
user_data = 0x5608fd464038
},
[47] = {
pos = -1064358253,
events = 32732,
fd = -45983008,
user_data = 0x5608fd4216c0
},
[48] = {
pos = 15,
events = 0,
fd = 338915584,
user_data = 0x7ffd3fb44190
},
[49] = {
pos = -128,
events = 4294967295,
fd = 0,
user_data = 0x0
},
[50] = {
pos = 0,
events = 0,
fd = -45721624,
user_data = 0x1
},
[51] = {
pos = -1064600273,
events = 32732,
fd = -45983008,
user_data = 0x96a2bfb414337100
},
[52] = {
pos = 1068778200,
events = 32765,
fd = -128,
user_data = 0x400
},
[53] = {
pos = -1064687279,
events = 32732,
fd = -1064692848,
user_data = 0x0
},
[54] = {
pos = 0,
events = 0,
fd = 0,
user_data = 0x0
},
[55] = {
pos = 338915584,
events = 2527248308,
fd = -45893664,
user_data = 0x96a2bfb414337100
},
[56] = {
pos = 0,
events = 0,
fd = 338915584,
user_data = 0x13
},
[57] = {
pos = -1064980103,
events = 32732,
fd = -45721624,
user_data = 0x5608fc77a0e9 <fork_process+216 at fork_process.c:123>
},
[58] = {
pos = 4194304,
events = 0,
fd = -45721624,
user_data = 0x7ffd3fb44280
},
[59] = {
pos = -57215663,
events = 22024,
fd = -45536520,
user_data = 0x0
},
[60] = {
pos = 1068778144,
events = 32765,
fd = -60056065,
user_data = 0x5608fd4657e8
},
[61] = {
pos = 0,
events = 0,
fd = 1068778256,
user_data = 0x5608fc6bd2c5 <load_ident+334 at palloc.h:142>
},
[62] = {
pos = -46000448,
events = 22024,
fd = -45983008,
user_data = 0x5608fd463f20
},
[63] = {
pos = -59249829,
events = 22024,
fd = 0,
user_data = 0x0
}
}
nevents = 1
__func__ = "ServerLoop"
#21 0x00005608fc7811ee in PostmasterMain (argc=argc@entry=19,
argv=argv@entry=0x5608fd41d050) at postmaster.c:1463
opt = <optimized out>
status = <optimized out>
userDoption = <optimized out>
listen_addr_saved = true
i = <optimized out>
output_config_variable = <optimized out>
__func__ = "PostmasterMain"
#22 0x00005608fc6c405e in main (argc=19, argv=0x5608fd41d050) at main.c:200
do_check_root = <optimized out>
