I got a bit background for sigsegv case. The call stack from core:
core 'core' of 28557: /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386 000000000051af5e cast_expression () + 16 00000000005337b5 fake_return_assignment () + 169 0000000000535747 db_assign_return_states_callback () + 419 fffff7ffeeea7009 sqlite3_exec () + 559 000000000050939f sql_exec () + 1a1 000000000050eb68 sql_select_return_states () + 15f 00000000005357f4 db_return_states_assign () + a2 0000000000535b48 match_assign_call () + 91 000000000053bc8e pass_expr_to_client () + 1f 000000000053bdbf __pass_to_client () + c8 0000000000529a54 parse_assignment () + 187 0000000000529ede __split_expr () + 2ef 000000000052c28c __split_stmt () + 335 000000000052be7e split_if_statement () + 258 000000000052c2ae __split_stmt () + 357 000000000052bae7 split_compound () + 13f 000000000052c29d __split_stmt () + 346 000000000052be64 split_if_statement () + 23e 000000000052c2ae __split_stmt () + 357 000000000052ba72 split_compound () + ca 000000000052c29d __split_stmt () + 346 000000000052be64 split_if_statement () + 23e 000000000052c2ae __split_stmt () + 357 000000000052bae7 split_compound () + 13f 000000000052c29d __split_stmt () + 346 000000000052a8a6 handle_pre_loop () + 230 000000000052c33f __split_stmt () + 3e8 000000000052ba72 split_compound () + ca 000000000052c29d __split_stmt () + 346 000000000052de3e parse_fn_statements () + 24 000000000052e264 split_function () + 1c2 000000000052f08e split_c_file_functions () + 1d2 000000000052f44a smatch () + 17c 00000000004970a6 main () + 1f8 0000000000495827 _start_crt () + 87 0000000000495788 _start () + 18 now, running smatch with —debug does reveal some extra data: /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:209 nv_var_overload() set_state new [register_returns_early] 'return_ranges' 0-u32max mem-db: insert or ignore into hash_string values (0xd6338457fe83563, '../../../common/mdb/mdb_nv.c'); mem-db: insert into return_states values (0xd6338457fe83563, 'nv_var_overload', 18446735277306630320, 31, '0-u32max[$1]', 1, 0, -1, '209', 'struct mdb_var*(*)(struct mdb_var*, struct mdb_var*)'); mem-db: insert or ignore into hash_string values (0xd6338457fe83563, '../../../common/mdb/mdb_nv.c'); mem-db: insert into return_states values (0xd6338457fe83563, 'nv_var_overload', 18446735277306630320, 31, '0-u32max[$1]', 1, 1023, 1, '$', ''); mem-db: insert or ignore into hash_string values (0xd6338457fe83563, '../../../common/mdb/mdb_nv.c'); mem-db: insert into return_states values (0xd6338457fe83563, 'nv_var_overload', 18446735277306630320, 31, '0-u32max[$1]', 1, 103, 0, '$', '4096-ptr_max'); mem-db: insert or ignore into hash_string values (0xd6338457fe83563, '../../../common/mdb/mdb_nv.c'); mem-db: insert into return_states values (0xd6338457fe83563, 'nv_var_overload', 18446735277306630320, 31, '0-u32max[$1]', 1, 103, 0, '$->v_du.v_ndef->v_du.v_ndef', '0'); /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:209 nv_var_overload() set_state new [register_param_cleared] '*v' cleared mem-db: insert or ignore into hash_string values (0xd6338457fe83563, '../../../common/mdb/mdb_nv.c'); mem-db: insert into return_states values (0xd6338457fe83563, 'nv_var_overload', 18446735277306630320, 31, '0-u32max[$1]', 1, 2501, 0, '*$', ''); mem-db: insert or ignore into hash_string values (0xd6338457fe83563, '../../../common/mdb/mdb_nv.c'); mem-db: insert into return_states values (0xd6338457fe83563, 'nv_var_overload', 18446735277306630320, 31, '0-u32max[$1]', 1, 2525, 0, '$->v_du.v_ndef', '0-u32max[$1]'); mem-db: insert or ignore into hash_string values (0xd6338457fe83563, '../../../common/mdb/mdb_nv.c'); mem-db: insert into return_states values (0xd6338457fe83563, 'nv_var_overload', 18446735277306630320, 31, '0-u32max[$1]', 1, 1029, -1, '$', '== $1'); mem-db: insert or ignore into hash_string values (0xd6338457fe83563, '../../../common/mdb/mdb_nv.c'); mem-db: insert into return_states values (0xd6338457fe83563, 'nv_var_overload', 18446735277306630320, 31, '0-u32max[$1]', 1, 1028, -1, '$', '== $1'); mem-db: insert or ignore into hash_string values (0xd6338457fe83563, '../../../common/mdb/mdb_nv.c'); mem-db: insert into return_states values (0xd6338457fe83563, 'nv_var_overload', 18446735277306630320, 31, '0-u32max[$1]', 1, 1037, -1, '', '4'); mem-db: insert or ignore into hash_string values (0xd6338457fe83563, '../../../common/mdb/mdb_nv.c'); mem-db: insert or ignore into return_implies values (0xd6338457fe83563, 'nv_var_overload', 18446735277306630320, 1, 1068, -1, '', ''); mem-db: insert or ignore into hash_string values (0xd6338457fe83563, '../../../common/mdb/mdb_nv.c'); mem-db: insert or ignore into return_implies values (0xd6338457fe83563, 'nv_var_overload', 18446735277306630320, 1, 1006, 0, '$', '1'); mem-db: insert or ignore into hash_string values (0xd6338457fe83563, '../../../common/mdb/mdb_nv.c'); mem-db: insert or ignore into return_implies values (0xd6338457fe83563, 'nv_var_overload', 18446735277306630320, 1, 1047, 0, '', '0'); debug: select function, type, parameter, key, value from return_implies where call_id = '18446735277306630320'; /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() select function, type, parameter, key, value from return_implies where call_id = '18446735277306630320'; nv_var_interpos, 1068, -1, , nv_var_interpos, 1006, 2, $, 1 nv_var_interpos, 1006, 3, $, 1 nv_var_interpos, 1047, 0, , 0 nv_var_overload, 1068, -1, , nv_var_overload, 1006, 0, $, 1 nv_var_overload, 1047, 0, , 0 /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_smatch_extra] 'v' 4096-ptr_max => 4096-ptr_max debug: select function, type, parameter, key, value from return_implies where call_id = '18446735277306630320'; /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() select function, type, parameter, key, value from return_implies where call_id = '18446735277306630320'; nv_var_interpos, 1068, -1, , nv_var_interpos, 1006, 2, $, 1 nv_var_interpos, 1006, 3, $, 1 nv_var_interpos, 1047, 0, , 0 nv_var_overload, 1068, -1, , nv_var_overload, 1006, 0, $, 1 nv_var_overload, 1047, 0, , 0 /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_modification_hooks] 'v' v = nv->nv_hash[i] => v = nv_var_overload(v, w) /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_smatch_extra_links] 'v' v => undefined /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_comparison] 'nv->nv_hash[i] vs v' == => unknown /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_comparison_links] 'v' nv->nv_hash[i] vs v => undefined /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_ssa] 'v->v_du.v_ename' merged => undefined /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_ssa] 'v->v_lname' merged => undefined /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [check_deref] 'v' ok => ok /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_stored_conditions] 'v' true => undefined /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_stored_conditions] 'v->v_flags & 4' undefined => undefined /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_stored_conditions] 'v->v_flags & 16' undefined => undefined /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_stored_conditions] 'v->v_flags & 4' undefined => undefined /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_stored_conditions_links] 'v' v, v->v_flags & 4, v->v_flags & 16, v->v_flags & 4 => undefined /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_stored_conditions] 'v->v_flags & 4' undefined => undefined /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_stored_conditions] 'v->v_flags & 16' undefined => undefined /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_stored_conditions] 'v->v_flags & 4' undefined => undefined /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_stored_conditions_links] 'v->v_flags' v->v_flags & 4, v->v_flags & 16, v->v_flags & 4 => undefined debug: select distinct return from return_states where call_id = '18446735277306630320'; /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() select distinct return from return_states where call_id = '18446735277306630320'; 4096-ptr_max[$3] 0-u32max[$1] /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() parsing $3] /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() parsing $1] /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() val = 1-u32max remaining = ] /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state new [register_comparison] 'return fffff7ffed8750b0 vs v' == /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state new [register_comparison_links] 'return fffff7ffed8750b0' return fffff7ffed8750b0 vs v /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_comparison_links] 'v' undefined => return fffff7ffed8750b0 vs v /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_ssa] 'v->v_du.v_ename' undefined => undefined /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_smatch_extra] 'v->v_du.v_ename' 0-u32max => 0-u32max /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state new [register_param_bits_set] 'v->v_flags' 0x0 + 0xffffffffffffffff /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_bits] 'v->v_flags' 0x10 + 0xff => 0x0 + 0xffffffffffffffff /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_stored_conditions_links] 'v->v_flags' undefined => undefined /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_smatch_extra] 'v->v_flags' 16-255 => 0-255 /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_ssa] 'v->v_lname' undefined => undefined /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [register_assigned_expr] 'v->v_lname' nv->nv_hash[i]->v_lname => nv_var_overload(v, w)->v_lname debug: select return_id, return, type, parameter, key, value from return_states where call_id = '18446735277306630320' order by return_id, type; /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() select return_id, return, type, parameter, key, value from return_states where call_id = '18446735277306630320' order by return_id, type; 30, 4096-ptr_max[$3], 0, -1, 195, struct mdb_var*(*)(struct mdb_nv*, uint, struct mdb_var*, struct mdb_var*) 30, 4096-ptr_max[$3], 103, 3, $, 4096-ptr_max 30, 4096-ptr_max[$3], 103, 3, $, 4096-ptr_max 30, 4096-ptr_max[$3], 103, 3, $, 4096-ptr_max 30, 4096-ptr_max[$3], 1001, -1, $->v_du.v_ndef, 4096-ptr_max 30, 4096-ptr_max[$3], 1004, 2, $, 30, 4096-ptr_max[$3], 1023, 2, $, 30, 4096-ptr_max[$3], 1023, 3, $, 30, 4096-ptr_max[$3], 1028, -1, $, == $3 30, 4096-ptr_max[$3], 1029, -1, $, == $3 30, 4096-ptr_max[$3], 1037, -1, , 8 30, 4096-ptr_max[$3], 1051, 2, $->v_flags, 0x10 30, 4096-ptr_max[$3], 2525, 2, $->v_next, 0 30, 4096-ptr_max[$3], 2525, 3, $->v_du.v_ndef, 4096-ptr_max[$2] 30, 4096-ptr_max[$3], 2525, 3, $->v_next, 0-u32max 31, 0-u32max[$1], 0, -1, 209, struct mdb_var*(*)(struct mdb_var*, struct mdb_var*) 31, 0-u32max[$1], 103, 0, $, 4096-ptr_max 31, 0-u32max[$1], 103, 0, $->v_du.v_ndef->v_du.v_ndef, 0 31, 0-u32max[$1], 1023, 1, $, 31, 0-u32max[$1], 1028, -1, $, == $1 31, 0-u32max[$1], 1029, -1, $, == $1 31, 0-u32max[$1], 1037, -1, , 4 31, 0-u32max[$1], 2501, 0, *$, 31, 0-u32max[$1], 2525, 0, $->v_du.v_ndef, 0-u32max[$1] /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() parsing $3] /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state change [internal] 'unnull_path' true => true /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() parsing $3] /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() parsing $3] /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() parsing $3] /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() parsing $3] /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() set_state new [register_smatch_extra] 'v->v_du.v_ndef' 4096-ptr_max /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() parsing $3] /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() parsing $3] /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() parsing $3] /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() parsing $3] /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch: ../../../common/mdb/mdb_nv.c:295 mdb_nv_insert() parsing $3] Segmentation Fault (core dumped) /code/illumos-gate/usr/src/tools/proto/root_i386-nd/opt/onbld/bin/i386/smatch --debug -fident -finline -fno-inline-functions -fno-builtin -fno-asm -fdiagnostics-show-option -nodefaultlibs -D__sun -O -m32 -Wall -Wextra -Werror -Wno-missing-braces -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -Wno-array-bounds -p=illumos_user --disable=uninitialized,check_check_deref -Wno-vla -Wno-one-bit-signed-bitfield -Wno-external-function-has-definition -Wno-old-style-definition -Wno-strict-prototypes --fatal-checks --timeout=0 -Wno-maybe-uninitialized -Wno-char-subscripts -Wno-clobbered -Wno-parentheses -Wno-unused-variable -std=gnu99 -fno-inline-small-functions -fno-inline-functions-called-once -fno-ipa-cp -fno-ipa-icf -fno-clone-functions -fno-reorder-functions -fno-reorder-blocks-and-partition -fno-aggressive-loop-optimizations --param=max-inline-insns-single=450 -fstack-protector-strong -g -gdwarf-4 -gstrict-dwarf -std=gnu99 -DTEXT_DOMAIN="SUNW_OST_OSCMD" -D_TS_ERRNO -I/code/illumos-gate/proto/root_i386/usr/include -D_MDB -I. -I../.. -I../../../common -I../../mdb -c ../../../common/mdb/mdb_nv.c -o /tmp/cw.I4aGW3/cwK4a4W3.o tsoome@balrog:/code/illumos-gate/usr/src/cmd/mdb/intel/ia32/mdb$ As I understand, mdb_nv.c:295 is the location of source where smatch gets crash, and the lines with ‘parsing $3’ are about processing the sql query some lines above, so that we have processed 10 lines from query result, and it seems we do crash on 11th: 30, 4096-ptr_max[$3], 1037, -1, , 8 is it the missing value there which ends up in call stack as NULL pointer for expression? rgds, toomas
