Re: Samba dump (useless) core
On 5/18/19 4:14 PM, Andrea Venturoli wrote: I'm not saying it's solved, however, since these dumps in the past have stopped appearing several times, and later started happening again; so I'll have to keep an eye on this. Ok, so new dumps have allowed to better pinpoint where the overflow happens and write a 20 line C program to deterministically reproduce the problem. This is *NOT* in Samba code, but in FreeBSD base system! So the next question: I would like to step into libc (or other base libraries) functions with GDB. How do I do that? I have src and debug libraries installed, e.g. /usr/lib/debug/lib/libc.so.7.debug. GDB sees those files, but still won't step into them. Also, if I issue "file /usr/lib/debug/lib/libc.so.7.debug", I get "/usr/lib/debug/lib/libc.so.7.debug: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), corrupted program header size, with debug_info, not stripped". Is that "corrupted program header size" normal? bye & Thanks av. ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Samba dump (useless) core
On 5/8/19 1:44 PM, Jan Beich wrote:
Hello.
While I am by no means at the end of this, I'll post some updates.
> "stack overflow detected" messages are part of -fstack-protector*.
> Recently, ports/ default changed to -strong in order to follow base/.
>
> https://svnweb.freebsd.org/changeset/ports/499897
Reading the description of -fstack-protector vs
-fstack-protector-strong, it shouldn't make any difference in this function.
I tried "SSP_CFLAGS?= -fstack-protector" for confirmation and the canary
is still there.
So that changeset cannot be the culprit.
I do no dare trying "SSP_UNSAFE= yes", because, in case it's really a
buffer overflow, I prefer a core to undefined behaviour on production
servers!
Perhaps it's lack of imagination on my part, but I fail to see how SSP
could give a false positive...
Besides there are other hints which make me suspect a buffer overflow is
really happening.
> If neither DEBUG port option nor WITH_DEBUG make variable provide symbols
> then file a bug.
The DEBUG option is there and works (using base GDB had me think
otherwise, but after switching to devel/gdb I can see debug info).
However, ports seem to compile with -O2, which sometimes makes it hard
to unterstand what's going on; so I added "CFLAGS=
-O0" to Makefile.
Putting that binary (with -O0) in production on one server has halted
the stream of cores, at least for now: I had about one core per minute
before; now it has been working fine for 30 minutes.
The other server which shows this problem is not dumping cores right now.
I'm not saying it's solved, however, since these dumps in the past have
stopped appearing several times, and later started happening again; so
I'll have to keep an eye on this.
(My wild guess, but I'm not sure, is that this happens when a client
needs to change its trust password: it tries again and again, and if it
eventually succeeds, then I don't get any more core until another client
needs this).
> In addition to debug symbols try building with -fsanitize=address in
> order to save time finding a place where the overflow occurs.
I wanted to try this, but ended up with an error during "make configure":
Checking simple C program
==>
#define CONFIG_H_IS_FROM_SAMBA 1
#define _SAMBA_BUILD_ 4
#define HAVE_CONFIG_H 1
#define SRCDIR "/usr/ports/net/samba48/work/samba-4.8.12"
#define HAVE_SYS_UTSNAME_H 1
/* #undef SYSTEM_UNAME_SYSNAME */
/* #undef SYSTEM_UNAME_MACHINE */
/* #undef SYSTEM_UNAME_RELEASE */
/* #undef SYSTEM_UNAME_VERSION */
#define HAVE_STDIO_H 1
#include
int main(void) { printf("hello world"); return 0; }
<==
[1/2] Compiling test.c
['cc', '-O2', '-pipe', '-fsanitize=address', '-DLIBICONV_PLUG',
'-fno-color-diagnostics', '-fstack-protector-strong',
'-DLDAP_DEPRECATED', '-isystem', '/usr/local/include',
'-fno-strict-aliasing', '-fno-omit-frame-pointer',
'-I/usr/ports/net/samba48/work/samba-4.8.12/lib/replace/../../buildtools/wafsamba',
'-MD', '-Idefault', '-I..', '-Idefault', '-I..', '-DLIBICONV_PLUG',
'-isystem', '/usr/local/include', '-D_SAMBA_BUILD_=4',
'-DHAVE_CONFIG_H=1', '../test.c', '-c', '-o', 'default/test_1.o']
[2/2] Linking default/testprog
default/test_1.o: In function `asan.module_ctor':
test.c:(.text+0x22): undefined reference to `__asan_init'
test.c:(.text+0x27): undefined reference to
`__asan_version_mismatch_check_v8'
test.c:(.text+0x36): undefined reference to `__asan_register_globals'
default/test_1.o: In function `asan.module_dtor':
test.c:(.text+0x4c): undefined reference to `__asan_unregister_globals'
cc: error: linker command failed with exit code 1 (use -v to see invocation)
['cc', 'default/test_1.o', '-o',
'/usr/ports/net/samba48/work/samba-4.8.12/bin/.conf_check_0/testbuild/default/testprog',
'-fstack-protector-strong', '-L/usr/local/lib']
bye & Thanks
av.
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Samba dump (useless) core
On 5/8/19 2:46 PM, Hans Petter Selasky wrote: On 2019-05-08 13:44, Jan Beich wrote: Andrea Venturoli writes: Try to install GDB from ports and use that. It often helps. Right!!! Thanks!!! I always install installing it when I develop, but forgot to think about it in this sysadmin-side problem :) At least now the trace has some info: (gdb) bt #0 0x000803e0d98a in kill () from /lib/libc.so.7 #1 0x000803e0d940 in __fail (msg=) at /usr/src/lib/libc/secure/stack_protector.c:110 #2 0x000803e0d8b0 in __stack_chk_fail () at /usr/src/lib/libc/secure/stack_protector.c:117 #3 0x000801f274e3 in E_deshash (passwd=, p16=0x80e8b1530 "?I\246k\316@\323-\366b\017\300\277P\227L0\324~.\b") at ../libcli/auth/smbencrypt.c:153 #4 0x00081a6a5f17 in setup_given_passwords (io=0x7fffd940, g=0x7fffd980) at ../source4/dsdb/samdb/ldb_modules/password_hash.c:2379 #5 0x00081a6a314c in setup_password_fields (io=0x7fffd940) at ../source4/dsdb/samdb/ldb_modules/password_hash.c:2414 #6 0x00081a6a1cdd in password_hash_mod_do_mod (ac=) at ../source4/dsdb/samdb/ldb_modules/password_hash.c:4532 #7 get_domain_data_callback (req=, ares=) at ../source4/dsdb/samdb/ldb_modules/password_hash.c:3920 #8 0x00081d54457e in ltdb_callback (ev=, te=, t=..., private_data=) at ../lib/ldb/ldb_tdb/ldb_tdb.c:1654 #9 0x000803b1bbce in tevent_common_invoke_timer_handler () from /usr/local/lib/libtevent.so.0 #10 0x000803b1bd06 in tevent_common_loop_timer_delay () from /usr/local/lib/libtevent.so.0 #11 0x000803b1923f in ?? () from /usr/local/lib/libtevent.so.0 #12 0x000803b15ead in _tevent_loop_once () from /usr/local/lib/libtevent.so.0 #13 0x0008093b509b in ldb_wait (handle=0x823cea820, type=) at ../lib/ldb/common/ldb.c:639 #14 0x0008095d3c0f in dsdb_autotransaction_request (sam_ldb=0x80e85e1a0, req=0x823dc7de0) at ../source4/dsdb/common/util.c:1120 #15 samdb_set_password_internal (ldb=, mem_ctx=, user_dn=, domain_dn=, new_password=, lmNewHash=, ntNewHash=, lmOldHash=, ntOldHash=, reject_reason=, _dominfo=, permit_interdomain_trust=) at ../source4/dsdb/common/util.c:2340 #16 0x0008095d498d in samdb_set_password_sid (ldb=0x80e85e1a0, mem_ctx=, user_sid=, new_version=0xe8ef0607fff, new_password=0x7fffdec0, lmNewHash=0x0, ntNewHash=0x0, lmOldHash=0x0, ntOldHash=0x80e8b23a0, reject_reason=0x0, _dominfo=0x0) at ../source4/dsdb/common/util.c:2817 #17 0x000816d7eacd in dcesrv_netr_ServerPasswordSet2 (dce_call=, mem_ctx=, r=) at ../source4/rpc_server/netlogon/dcerpc_netlogon.c:821 #18 netlogon__op_dispatch (dce_call=0x823f62260, mem_ctx=0x823f62260, r=0x823cea6e0) at default/librpc/gen_ndr/ndr_netlogon_s.c:399 #19 0x000816d78241 in dcesrv_request (call=) at ../source4/rpc_server/dcerpc_server.c:1874 #20 dcesrv_process_ncacn_packet (dce_conn=, pkt=, blob=...) at ../source4/rpc_server/dcerpc_server.c:2196 #21 dcesrv_read_fragment_done (subreq=) at ../source4/rpc_server/dcerpc_server.c:2774 #22 0x00080682117a in dcerpc_read_ncacn_packet_done (subreq=) at ../librpc/rpc/dcerpc_util.c:835 #23 0x000806600619 in tstream_readv_pdu_readv_done (subreq=0x823ca6c80) at ../lib/tsocket/tsocket_helpers.c:319 #24 0x0008065ff7ce in tstream_readv_done (subreq=0x80e846700) at ../lib/tsocket/tsocket.c:604 #25 0x000803b16fa7 in tevent_common_invoke_immediate_handler () from /usr/local/lib/libtevent.so.0 #26 0x000803b17004 in tevent_common_loop_immediate () from /usr/local/lib/libtevent.so.0 #27 0x000803b1922c in ?? () from /usr/local/lib/libtevent.so.0 #28 0x000803b15ead in _tevent_loop_once () from /usr/local/lib/libtevent.so.0 #29 0x000803b160eb in tevent_common_loop_wait () from /usr/local/lib/libtevent.so.0 #30 0x00080ec09aeb in standard_accept_connection (ev=0x80e85e060, lp_ctx=0x80e853060, sock=, new_conn=0x8021355f0 , private_data=0x80e9db7c0, process_context=0x80e8ce860) at ../source4/smbd/process_standard.c:384 #31 0x000803b16b6d in tevent_common_invoke_fd_handler () from /usr/local/lib/libtevent.so.0 #32 0x000803b1990a in ?? () from /usr/local/lib/libtevent.so.0 #33 0x000803b15ead in _tevent_loop_once () from /usr/local/lib/libtevent.so.0 #34 0x000803b160eb in tevent_common_loop_wait () from /usr/local/lib/libtevent.so.0 #35 0x00080ec09d51 in standard_new_task (ev=0x80e85e060, lp_ctx=0x80e853060, service_name=0x816b4cff6 "rpc", new_task=0x802136200 , private_data=0x80e8cea90, service_details=0x80e84ecc0, from_parent_fd=24) at ../source4/smbd/process_standard.c:494 #36 0x0008021361e2 in task_server_startup (event_ctx=0x80e85e060, lp_ctx=0x80e853060, service_name=0x816b4cff6 "rpc", model_ops=0x80ee0bb90 , task_init=0x816b4cd70 , service_details=0x80e84ecc0, from_parent_fd=24) at ../source4/smbd/service_task.c:123 #37 0x000802134c36 in server_service_init (name=0x80e84c950 "rpc", event_context=, lp_ctx=, model_ops=, from_parent_fd=) at
Re: Samba dump (useless) core
On 2019-05-08 13:44, Jan Beich wrote: Andrea Venturoli writes: Try to install GDB from ports and use that. It often helps. --HPS ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Samba dump (useless) core
Andrea Venturoli writes: > While this sheds some light (seems the whole thing has to do with > trust password change), it shows a stack overflow, not an assert. "stack overflow detected" messages are part of -fstack-protector*. Recently, ports/ default changed to -strong in order to follow base/. https://svnweb.freebsd.org/changeset/ports/499897 > a) how do I get out of this? Remove the other member from the domain > and rejoin? Try and fix the DB (how)? Other? (I understand this would > be a question better answere on Samba list, but we are already here). Try adding either to the port's Makefile: SSP_CFLAGS?= -fstack-protector # XXX -strong crashes in some cases or SSP_UNSAFE= yes # XXX crashes in some cases > b) if there's a stack overflow, isn't that a bug? How can I check it > and report it (to FreeBSD or Samba, once the details can let me > decide) if I cannot get a proper core? > I guess I'd have to look into a way to generate debug info with > Poudriere... what's the proper way to do this? How debugging is implemented is up to port maintainer but the framework provides WITH_DEBUG facility that *usually* allows to get debug symbols. If neither DEBUG port option nor WITH_DEBUG make variable provide symbols then file a bug. https://www.freebsd.org/doc/en/books/porters-handbook/install.html#install-strip In addition to debug symbols try building with -fsanitize=address in order to save time finding a place where the overflow occurs. ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Samba dump (useless) core
On 5/8/19 12:25 PM, Konstantin Belousov wrote: Signal 6 is SIGABRT, which means most likely that some assert was triggered. You should look into your logs. samba.log shows nothing at level 1 (default); at level 3 gives gobs of information and I don't look what to look for. In all.log, however, I have something like this: May 8 12:48:11 winbindd[83147]: [2019/05/08 12:48:11.493769, 0] ../source3/libsmb/trusts_util.c:334(trust_pw_change) May 8 12:48:11 kernel: May 8 12:48:09 last message repeated 4 times May 8 12:48:11 kernel: May 8 12:48:11 winbindd[83147]: [2019/05/08 12:48:11.493769, 0] ../source3/libsmb/trusts_util.c:334(trust_pw_change) May 8 12:48:11 winbindd[83147]: 2019/05/08 12:48:11 : trust_pw_change(XX): A password change was already started against 'dc1.ad.x.it' at Mon Apr 29 12:22:51 2019 CEST. Trying to recover... May 8 12:48:11 kernel: May 8 12:48:11 winbindd[83147]: 2019/05/08 12:48:11 : trust_pw_change(XX): A password change was already started against 'dc1.ad.x.it' at Mon Apr 29 12:22:51 2019 CEST. Trying to recover... May 8 12:48:11 winbindd[83147]: [2019/05/08 12:48:11.493910, 0] ../source3/libsmb/trusts_util.c:343(trust_pw_change) May 8 12:48:11 kernel: May 8 12:48:11 winbindd[83147]: [2019/05/08 12:48:11.493910, 0] ../source3/libsmb/trusts_util.c:343(trust_pw_change) May 8 12:48:11 winbindd[83147]: 2019/05/08 12:48:11 : trust_pw_change(XX): Last failure local[NT_STATUS_NOT_COMMITTED] remote[NT_STATUS_CONNECTION_DISCONNECTED] against 'dc1.ad.x.it' at Wed May 8 12:47:11 2019 CEST. May 8 12:48:11 kernel: May 8 12:48:11 winbindd[83147]: 2019/05/08 12:48:11 : trust_pw_change(XX): Last failure local[NT_STATUS_NOT_COMMITTED] remote[NT_STATUS_CONNECTION_DISCONNECTED] against 'dc1.ad.x.it' at Wed May 8 12:47:11 2019 CEST. May 8 12:48:11 winbindd[83147]: [2019/05/08 12:48:11.493986, 0] ../source3/libsmb/trusts_util.c:380(trust_pw_change) May 8 12:48:11 kernel: May 8 12:48:11 winbindd[83147]: [2019/05/08 12:48:11.493986, 0] ../source3/libsmb/trusts_util.c:380(trust_pw_change) May 8 12:48:11 winbindd[83147]: 2019/05/08 12:48:11 : trust_pw_change(XX): Verifying passwords remotely netlogon_creds_cli:CLI[FS/FS$]/SRV[DC1/XX]. May 8 12:48:11 kernel: May 8 12:48:11 winbindd[83147]: 2019/05/08 12:48:11 : trust_pw_change(XX): Verifying passwords remotely netlogon_creds_cli:CLI[FS/FS$]/SRV[DC1/XX]. May 8 12:48:11 samba[54713]: stack overflow detected; terminated May 8 12:48:11 winbindd[83147]: [2019/05/08 12:48:11.496724, 0] ../source3/libsmb/trusts_util.c:452(trust_pw_change) May 8 12:48:11 kernel: May 8 12:48:11 samba[54713]: stack overflow detected; terminated May 8 12:48:11 kernel: May 8 12:48:11 winbindd[83147]: [2019/05/08 12:48:11.496724, 0] ../source3/libsmb/trusts_util.c:452(trust_pw_change) May 8 12:48:11 winbindd[83147]: 2019/05/08 12:48:11 : trust_pw_change(XX): Verified old password remotely using netlogon_creds_cli:CLI[FS/FS$]/SRV[DC1/XX] May 8 12:48:11 kernel: May 8 12:48:11 winbindd[83147]: 2019/05/08 12:48:11 : trust_pw_change(XX): Verified old password remotely using netlogon_creds_cli:CLI[FS/FS$]/SRV[DC1/XX] May 8 12:48:11 winbindd[83147]: [2019/05/08 12:48:11.496828, 0] ../source3/libsmb/trusts_util.c:491(trust_pw_change) May 8 12:48:11 kernel: May 8 12:48:11 winbindd[83147]: [2019/05/08 12:48:11.496828, 0] ../source3/libsmb/trusts_util.c:491(trust_pw_change) May 8 12:48:11 winbindd[83147]: 2019/05/08 12:48:11 : trust_pw_change(XX): Changed password locally May 8 12:48:11 kernel: May 8 12:48:11 winbindd[83147]: 2019/05/08 12:48:11 : trust_pw_change(XX): Changed password locally May 8 12:48:11 winbindd[83147]: [2019/05/08 12:48:11.636003, 0] ../source3/libsmb/trusts_util.c:507(trust_pw_change) May 8 12:48:11 kernel: pid 54713 (samba), uid 0: exited on signal 6 (core dumped) May 8 12:48:11 kernel: May 8 12:48:11 winbindd[83147]: [2019/05/08 12:48:11.636003, 0] ../source3/libsmb/trusts_util.c:507(trust_pw_change) May 8 12:48:11 winbindd[83147]: 2019/05/08 12:48:11 : trust_pw_change(XX) remote password change with netlogon_creds_cli:CLI[FS/FS$]/SRV[DC1/XX] failed - NT_STATUS_CONNECTION_DISCONNECTED (disconnected) May 8 12:48:11 kernel: May 8 12:48:11 winbindd[83147]: 2019/05/08 12:48:11 : trust_pw_change(XX) remote password change with netlogon_creds_cli:CLI[FS/FS$]/SRV[DC1/XX] failed - NT_STATUS_CONNECTION_DISCONNECTED (disconnected ) May 8 12:48:11 samba[80265]: [2019/05/08 12:48:11.639630, 0] ../source4/smbd/process_standard.c:158(standard_child_pipe_handler) May 8 12:48:11 kernel: May 8 12:48:11 samba[80265]: [2019/05/08 12:48:11.639630, 0]
Re: Samba dump (useless) core
On Wed, May 08, 2019 at 12:03:34PM +0200, Andrea Venturoli wrote: > Hello. > > I've got a few servers where I run Samba 4.8 in a jail as an AD DC. > > Lately, on a couple of them, it started dumping core: on one server, > apparently everything works fine after that; on another one I have > intermittent DNS issues, but I'm not sure they are related to this core. > > I tried examining the core, but this is what I get: > > > # gdb /usr/local/sbin/samba samba.core > > GNU gdb 6.1.1 [FreeBSD] > > Copyright 2004 Free Software Foundation, Inc. > > GDB is free software, covered by the GNU General Public License, and you are > > welcome to change it and/or distribute copies of it under certain > > conditions. > > Type "show copying" to see the conditions. > > There is absolutely no warranty for GDB. Type "show warranty" for details. > > This GDB was configured as "amd64-marcel-freebsd"... > > Core was generated by `samba: conn[rpc] c[ipv4:192.168.xxx.4:31803] > > s[ipv4:192.168.134.3:49153] server_'. > > Program terminated with signal 6, Aborted. > > #0 0x000803e0d98a in ?? () > > (gdb) info thr > > * 1 process 101460 0x000803e0d98a in ?? () > > (gdb) bt > > #0 0x000803e0d98a in ?? () > > #1 0x000803e0d940 in ?? () > > #2 0xffdf in ?? () > > #3 0x in ?? () > > #4 0x in ?? () > > (gdb) > > The only relevant line, unfortunately is "Core was generated by...", but > it doesn't help much. > > Samba is compiled (via Pudriere) with DEBUG, so I thought the above > would be more useful. > > Any hint? Signal 6 is SIGABRT, which means most likely that some assert was triggered. You should look into your logs. Also it is possible that DEBUG turns on asserts. ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Samba dump (useless) core
Hello. I've got a few servers where I run Samba 4.8 in a jail as an AD DC. Lately, on a couple of them, it started dumping core: on one server, apparently everything works fine after that; on another one I have intermittent DNS issues, but I'm not sure they are related to this core. I tried examining the core, but this is what I get: # gdb /usr/local/sbin/samba samba.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Core was generated by `samba: conn[rpc] c[ipv4:192.168.xxx.4:31803] s[ipv4:192.168.134.3:49153] server_'. Program terminated with signal 6, Aborted. #0 0x000803e0d98a in ?? () (gdb) info thr * 1 process 101460 0x000803e0d98a in ?? () (gdb) bt #0 0x000803e0d98a in ?? () #1 0x000803e0d940 in ?? () #2 0xffdf in ?? () #3 0x in ?? () #4 0x in ?? () (gdb) The only relevant line, unfortunately is "Core was generated by...", but it doesn't help much. Samba is compiled (via Pudriere) with DEBUG, so I thought the above would be more useful. Any hint? bye & Thanks av. ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
