Re: segfault with stripped lib, works fine when non-stripped
Hey, On Tue, Jan 5, 2016 at 7:38 AM, Stuart Henderson wrote: > On 2016-01-04, Jeremie Le Hen wrote: >> Hi, >> >> Yeah... when you read that subject you probably had this weird gaze ô_Ò >> like I did when I came to that conclusion. >> >> I've been experiencing segfaults in milter-greylist on one of my MX >> running OpenBSD for a while. I contacted Stuart (cc'ed) about 6 months >> ago about this, but gave up because I couldn't manage to compile >> everything with the debugging symbols. This time after much struggle to >> compile the ports chain with them, I finally managed to run >> milter-greylist in gdb(1) with the hope to witness the live crash and >> get a detailed stacktrace... >> >> Except that even after tinkling Postfix, it never happened. This simply >> worked fine. So after some more tinkering I came to the following >> conclusion: if I run strip(1) on /usr/local/lib/libbind/libbind.so.5.0 >> to remove the debugging symbols, then it will crash with the stacktrace >> below. > > libbind in the package isn't stripped either, it's just that it isn't built > with debug symbols. So I'm not sure what's going on there. > > Note for anyone else looking: this is all rather dirty because there are > conflicts between symbols in libc and libbind. It works well enough for net/mtr > and for the test program spf_example in libspf2's distribution, but I suspect > using it as a milter in the address space of an MTA that's using the libc > resolver is pushing our luck too far. Alright, I finally got some time (vacations) to fix this. I got rid of the problem by removing the actually useless dependency over libbind. The binary has been running for one hour without crashing which was impossible previously. See the following patches for mail/libspf2 and mail/milter-greylist (I cc'ed Jakob, the maintainer): https://people.freebsd.org/~jlh/openbsd_mail_libspf2_no_libbind.diff https://people.freebsd.org/~jlh/openbsd_mail_milter-greylist_no_libbind.diff > > >> #0 0x1cc53e386d40 in memcpy (dst0=0x1cc5c48b7000, src0=Variable "src0" is not available. >> ) at /usr/src/lib/libc/string/memcpy.c:94 >> #1 0x1cc4f4d496d8 in __res_vinit () from /usr/local/lib/libbind/libbind.so.5.0 >> #2 0x1cc4f4d48bda in __res_ninit () from /usr/local/lib/libbind/libbind.so.5.0 >> #3 0x1cc50b181905 in SPF_dns_resolv_lookup (spf_dns_server=0x1cc5c48ab780, domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at spf_dns_resolv.c:261 >> #4 0x1cc50b180117 in SPF_dns_lookup (spf_dns_server=0x1cc5c48ab780, domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at spf_dns.c:141 >> #5 0x1cc50b180b16 in SPF_dns_cache_lookup (spf_dns_server=0x1cc5c48abc80, domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at spf_dns_cache.c:408 >> #6 0x1cc50b180117 in SPF_dns_lookup (spf_dns_server=0x1cc5c48abc80, domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at spf_dns.c:141 >> #7 0x1cc50b18e4e3 in SPF_server_get_record (spf_server=0x1cc5eb4154c0, spf_request=0x1cc5c48aeb00, spf_response=0x1cc5eb41b400, spf_recordp=0x1cc54f7c8700) at spf_server.c:351 >> #8 0x1cc50b18c959 in SPF_request_query_mailfrom (spf_request=0x1cc5c48aeb00, spf_responsep=0x1cc54f7c87a0) at spf_request.c:291 >> #9 0x1cc2ee1207ca in spf_check_internal (ad=0x1cc4f4c65948, as=AS_RCPT, ap=0x1cc54f7c8cd0, priv=0x1cc5c48af000) at spf.c:388 >> #10 0x1cc2ee120c17 in spf_check (ad=0x1cc4f4c65948, as=AS_RCPT, ap=0x1cc54f7c8cd0, priv=0x1cc5c48af000) at spf.c:524 >> #11 0x1cc2ee123a0d in acl_filter (stage=AS_RCPT, ctx=0x1cc5c48b2000, priv=0x1cc5c48af000) at acl.c:1902 >> #12 0x1cc2ee1069ae in real_envrcpt (ctx=0x1cc5c48b2000, envrcpt=0x1cc5eb41c280) at milter-greylist.c:601 >> #13 0x1cc2ee105de0 in mlfi_envrcpt (ctx=0x1cc5c48b2000, envrcpt=0x1cc5eb41c280) at milter-greylist.c:213 >> #14 0x1cc52bfaa46e in st_rcpt () from /usr/local/lib/libmilter.so.4.0 >> #15 0x1cc52bfab557 in mi_engine () from /usr/local/lib/libmilter.so.4.0 >> #16 0x1cc52bfaca10 in mi_handle_session () from /usr/local/lib/libmilter.so.4.0 >> #17 0x1cc52bfab7d9 in mi_thread_handle_wrapper () from /usr/local/lib/libmilter.so.4.0 >> #18 0x1cc5a247d90e in _rthread_start (v=Variable "v" is not available. >> ) at /usr/src/lib/librthread/rthread.c:145 >> #19 0x1cc53e33649b in __tfork_thread () at /usr/src/lib/libc/arch/amd64/sys/tfork_thread.S:75 >> #20 0x in ?? () > -- Jeremie Le Hen j...@freebsd.org
Re: segfault with stripped lib, works fine when non-stripped
Op Mon, 04 Jan 2016 22:54:24 +0100 schreef Jeremie Le Hen : Yeah... when you read that subject you probably had this weird gaze ô_Ò like I did when I came to that conclusion. I've been experiencing segfaults in milter-greylist [...] So after some more tinkering I came to the following conclusion: if I run strip(1) on /usr/local/lib/libbind/libbind.so.5.0 to remove the debugging symbols, then it will crash with the stacktrace below. Has anyone of you seen such a behavior in the past? #0 0x1cc53e386d40 in memcpy (dst0=0x1cc5c48b7000, src0=Variable "src0" is not available. ) at /usr/src/lib/libc/string/memcpy.c:94 #1 0x1cc4f4d496d8 in __res_vinit () from /usr/local/lib/libbind/libbind.so.5.0 #2 0x1cc4f4d48bda in __res_ninit () from /usr/local/lib/libbind/libbind.so.5.0 #3 0x1cc50b181905 in SPF_dns_resolv_lookup (spf_dns_server=0x1cc5c48ab780, domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at spf_dns_resolv.c:261 [...] I had the same stack trace with a simple test application (http://www.libspf2.org/docs/html/ ). Except that my memcpy showed src=0, len=16. OpenBSD 5.8 i386 with stock libs (from packages). The prebuilt tools (in /usr/local/libexec/libspf2/) ran fine. My Makefile had: LDADD+= -lspf2 Segfaults disappeared when I changed that to: LDADD+= -lbind -lspf2 So yeah, looks like conflicts between symbols. -- (Remove the obvious prefix to reply privately.) Gemaakt met Opera's e-mailprogramma: http://www.opera.com/mail/
Re: segfault with stripped lib, works fine when non-stripped
On 2016-01-04, Jeremie Le Hen wrote: > Hi, > > Yeah... when you read that subject you probably had this weird gaze ô_Ò > like I did when I came to that conclusion. > > I've been experiencing segfaults in milter-greylist on one of my MX > running OpenBSD for a while. I contacted Stuart (cc'ed) about 6 months > ago about this, but gave up because I couldn't manage to compile > everything with the debugging symbols. This time after much struggle to > compile the ports chain with them, I finally managed to run > milter-greylist in gdb(1) with the hope to witness the live crash and > get a detailed stacktrace... > > Except that even after tinkling Postfix, it never happened. This simply > worked fine. So after some more tinkering I came to the following > conclusion: if I run strip(1) on /usr/local/lib/libbind/libbind.so.5.0 > to remove the debugging symbols, then it will crash with the stacktrace > below. libbind in the package isn't stripped either, it's just that it isn't built with debug symbols. So I'm not sure what's going on there. Note for anyone else looking: this is all rather dirty because there are conflicts between symbols in libc and libbind. It works well enough for net/mtr and for the test program spf_example in libspf2's distribution, but I suspect using it as a milter in the address space of an MTA that's using the libc resolver is pushing our luck too far. > #0 0x1cc53e386d40 in memcpy (dst0=0x1cc5c48b7000, src0=Variable "src0" > is not available. > ) at /usr/src/lib/libc/string/memcpy.c:94 > #1 0x1cc4f4d496d8 in __res_vinit () from > /usr/local/lib/libbind/libbind.so.5.0 > #2 0x1cc4f4d48bda in __res_ninit () from > /usr/local/lib/libbind/libbind.so.5.0 > #3 0x1cc50b181905 in SPF_dns_resolv_lookup > (spf_dns_server=0x1cc5c48ab780, domain=0x1cc55122c1d0 "mydomain.org", > rr_type=ns_t_spf, should_cache=1) at spf_dns_resolv.c:261 > #4 0x1cc50b180117 in SPF_dns_lookup (spf_dns_server=0x1cc5c48ab780, > domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at > spf_dns.c:141 > #5 0x1cc50b180b16 in SPF_dns_cache_lookup > (spf_dns_server=0x1cc5c48abc80, domain=0x1cc55122c1d0 "mydomain.org", > rr_type=ns_t_spf, should_cache=1) at spf_dns_cache.c:408 > #6 0x1cc50b180117 in SPF_dns_lookup (spf_dns_server=0x1cc5c48abc80, > domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at > spf_dns.c:141 > #7 0x1cc50b18e4e3 in SPF_server_get_record (spf_server=0x1cc5eb4154c0, > spf_request=0x1cc5c48aeb00, spf_response=0x1cc5eb41b400, > spf_recordp=0x1cc54f7c8700) at spf_server.c:351 > #8 0x1cc50b18c959 in SPF_request_query_mailfrom > (spf_request=0x1cc5c48aeb00, spf_responsep=0x1cc54f7c87a0) at > spf_request.c:291 > #9 0x1cc2ee1207ca in spf_check_internal (ad=0x1cc4f4c65948, as=AS_RCPT, > ap=0x1cc54f7c8cd0, priv=0x1cc5c48af000) at spf.c:388 > #10 0x1cc2ee120c17 in spf_check (ad=0x1cc4f4c65948, as=AS_RCPT, > ap=0x1cc54f7c8cd0, priv=0x1cc5c48af000) at spf.c:524 > #11 0x1cc2ee123a0d in acl_filter (stage=AS_RCPT, ctx=0x1cc5c48b2000, > priv=0x1cc5c48af000) at acl.c:1902 > #12 0x1cc2ee1069ae in real_envrcpt (ctx=0x1cc5c48b2000, > envrcpt=0x1cc5eb41c280) at milter-greylist.c:601 > #13 0x1cc2ee105de0 in mlfi_envrcpt (ctx=0x1cc5c48b2000, > envrcpt=0x1cc5eb41c280) at milter-greylist.c:213 > #14 0x1cc52bfaa46e in st_rcpt () from /usr/local/lib/libmilter.so.4.0 > #15 0x1cc52bfab557 in mi_engine () from /usr/local/lib/libmilter.so.4.0 > #16 0x1cc52bfaca10 in mi_handle_session () from > /usr/local/lib/libmilter.so.4.0 > #17 0x1cc52bfab7d9 in mi_thread_handle_wrapper () from > /usr/local/lib/libmilter.so.4.0 > #18 0x1cc5a247d90e in _rthread_start (v=Variable "v" is not available. > ) at /usr/src/lib/librthread/rthread.c:145 > #19 0x1cc53e33649b in __tfork_thread () at > /usr/src/lib/libc/arch/amd64/sys/tfork_thread.S:75 > #20 0x in ?? ()
Re: segfault with stripped lib, works fine when non-stripped
Hi Jeremie, On Mon, Jan 04, 2016 at 10:54:24PM +0100, Jeremie Le Hen wrote: > [...] > Has anyone of you seen such a behavior in the past? > [...] Haven't seen something like that but my next step would be to build it with CFLAGS="-g -O0" and without stripping for maximum debuggability and run it in valgrind to see if there are weird memory access patterns. -- Gregor
Re: segfault with stripped lib, works fine when non-stripped
On Tue, Jan 5, 2016 at 6:54 AM, Jeremie Le Hen wrote: > Hi, > > Yeah... when you read that subject you probably had this weird gaze ô_Ò > like I did when I came to that conclusion. Theo will probably get after me for responding when I don't know what I'm talking about, and I don't have specific experience with this package, but it's not unusual, in general. Stripping symbols can reveal lots of things where prorammers have cut corners or just made mistakes -- pointer issues are especially prominent but not the only problems. However, in this case, I'm guessing it would probably not be pointer issues so much as something like allocation class issues that are hidden when the symbols are left in, but exposed when the symbols are stripped. That said, you may not want to think too hard about that just yet. > I've been experiencing segfaults in milter-greylist on one of my MX > running OpenBSD for a while. I contacted Stuart (cc'ed) about 6 months > ago about this, but gave up because I couldn't manage to compile > everything with the debugging symbols. This time after much struggle to > compile the ports chain with them, I finally managed to run > milter-greylist in gdb(1) with the hope to witness the live crash and > get a detailed stacktrace... > > Except that even after tinkling Postfix, it never happened. This simply > worked fine. So after some more tinkering I came to the following > conclusion: if I run strip(1) on /usr/local/lib/libbind/libbind.so.5.0 > to remove the debugging symbols, then it will crash with the stacktrace > below. > > Has anyone of you seen such a behavior in the past? > > #0 0x1cc53e386d40 in memcpy (dst0=0x1cc5c48b7000, src0=Variable "src0" is not available. Do you have any idea why "src0" isn't available here? That might be a good place to start. > ) at /usr/src/lib/libc/string/memcpy.c:94 > #1 0x1cc4f4d496d8 in __res_vinit () from /usr/local/lib/libbind/libbind.so.5.0 > #2 0x1cc4f4d48bda in __res_ninit () from /usr/local/lib/libbind/libbind.so.5.0 > #3 0x1cc50b181905 in SPF_dns_resolv_lookup (spf_dns_server=0x1cc5c48ab780, domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at spf_dns_resolv.c:261 > #4 0x1cc50b180117 in SPF_dns_lookup (spf_dns_server=0x1cc5c48ab780, domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at spf_dns.c:141 > #5 0x1cc50b180b16 in SPF_dns_cache_lookup (spf_dns_server=0x1cc5c48abc80, domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at spf_dns_cache.c:408 > #6 0x1cc50b180117 in SPF_dns_lookup (spf_dns_server=0x1cc5c48abc80, domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at spf_dns.c:141 > #7 0x1cc50b18e4e3 in SPF_server_get_record (spf_server=0x1cc5eb4154c0, spf_request=0x1cc5c48aeb00, spf_response=0x1cc5eb41b400, spf_recordp=0x1cc54f7c8700) at spf_server.c:351 > #8 0x1cc50b18c959 in SPF_request_query_mailfrom (spf_request=0x1cc5c48aeb00, spf_responsep=0x1cc54f7c87a0) at spf_request.c:291 > #9 0x1cc2ee1207ca in spf_check_internal (ad=0x1cc4f4c65948, as=AS_RCPT, ap=0x1cc54f7c8cd0, priv=0x1cc5c48af000) at spf.c:388 > #10 0x1cc2ee120c17 in spf_check (ad=0x1cc4f4c65948, as=AS_RCPT, ap=0x1cc54f7c8cd0, priv=0x1cc5c48af000) at spf.c:524 > #11 0x1cc2ee123a0d in acl_filter (stage=AS_RCPT, ctx=0x1cc5c48b2000, priv=0x1cc5c48af000) at acl.c:1902 > #12 0x1cc2ee1069ae in real_envrcpt (ctx=0x1cc5c48b2000, envrcpt=0x1cc5eb41c280) at milter-greylist.c:601 > #13 0x1cc2ee105de0 in mlfi_envrcpt (ctx=0x1cc5c48b2000, envrcpt=0x1cc5eb41c280) at milter-greylist.c:213 > #14 0x1cc52bfaa46e in st_rcpt () from /usr/local/lib/libmilter.so.4.0 > #15 0x1cc52bfab557 in mi_engine () from /usr/local/lib/libmilter.so.4.0 > #16 0x1cc52bfaca10 in mi_handle_session () from /usr/local/lib/libmilter.so.4.0 > #17 0x1cc52bfab7d9 in mi_thread_handle_wrapper () from /usr/local/lib/libmilter.so.4.0 > #18 0x1cc5a247d90e in _rthread_start (v=Variable "v" is not available. > ) at /usr/src/lib/librthread/rthread.c:145 > #19 0x1cc53e33649b in __tfork_thread () at /usr/src/lib/libc/arch/amd64/sys/tfork_thread.S:75 > #20 0x in ?? () > > > -- > Jeremie Le Hen > j...@freebsd.org > -- Joel Rees Be careful when you look at conspiracy. Arm yourself with knowledge of yourself, as well: http://reiisi.blogspot.jp/2011/10/conspiracy-theories.html
segfault with stripped lib, works fine when non-stripped
Hi, Yeah... when you read that subject you probably had this weird gaze ô_Ò like I did when I came to that conclusion. I've been experiencing segfaults in milter-greylist on one of my MX running OpenBSD for a while. I contacted Stuart (cc'ed) about 6 months ago about this, but gave up because I couldn't manage to compile everything with the debugging symbols. This time after much struggle to compile the ports chain with them, I finally managed to run milter-greylist in gdb(1) with the hope to witness the live crash and get a detailed stacktrace... Except that even after tinkling Postfix, it never happened. This simply worked fine. So after some more tinkering I came to the following conclusion: if I run strip(1) on /usr/local/lib/libbind/libbind.so.5.0 to remove the debugging symbols, then it will crash with the stacktrace below. Has anyone of you seen such a behavior in the past? #0 0x1cc53e386d40 in memcpy (dst0=0x1cc5c48b7000, src0=Variable "src0" is not available. ) at /usr/src/lib/libc/string/memcpy.c:94 #1 0x1cc4f4d496d8 in __res_vinit () from /usr/local/lib/libbind/libbind.so.5.0 #2 0x1cc4f4d48bda in __res_ninit () from /usr/local/lib/libbind/libbind.so.5.0 #3 0x1cc50b181905 in SPF_dns_resolv_lookup (spf_dns_server=0x1cc5c48ab780, domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at spf_dns_resolv.c:261 #4 0x1cc50b180117 in SPF_dns_lookup (spf_dns_server=0x1cc5c48ab780, domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at spf_dns.c:141 #5 0x1cc50b180b16 in SPF_dns_cache_lookup (spf_dns_server=0x1cc5c48abc80, domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at spf_dns_cache.c:408 #6 0x1cc50b180117 in SPF_dns_lookup (spf_dns_server=0x1cc5c48abc80, domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at spf_dns.c:141 #7 0x1cc50b18e4e3 in SPF_server_get_record (spf_server=0x1cc5eb4154c0, spf_request=0x1cc5c48aeb00, spf_response=0x1cc5eb41b400, spf_recordp=0x1cc54f7c8700) at spf_server.c:351 #8 0x1cc50b18c959 in SPF_request_query_mailfrom (spf_request=0x1cc5c48aeb00, spf_responsep=0x1cc54f7c87a0) at spf_request.c:291 #9 0x1cc2ee1207ca in spf_check_internal (ad=0x1cc4f4c65948, as=AS_RCPT, ap=0x1cc54f7c8cd0, priv=0x1cc5c48af000) at spf.c:388 #10 0x1cc2ee120c17 in spf_check (ad=0x1cc4f4c65948, as=AS_RCPT, ap=0x1cc54f7c8cd0, priv=0x1cc5c48af000) at spf.c:524 #11 0x1cc2ee123a0d in acl_filter (stage=AS_RCPT, ctx=0x1cc5c48b2000, priv=0x1cc5c48af000) at acl.c:1902 #12 0x1cc2ee1069ae in real_envrcpt (ctx=0x1cc5c48b2000, envrcpt=0x1cc5eb41c280) at milter-greylist.c:601 #13 0x1cc2ee105de0 in mlfi_envrcpt (ctx=0x1cc5c48b2000, envrcpt=0x1cc5eb41c280) at milter-greylist.c:213 #14 0x1cc52bfaa46e in st_rcpt () from /usr/local/lib/libmilter.so.4.0 #15 0x1cc52bfab557 in mi_engine () from /usr/local/lib/libmilter.so.4.0 #16 0x1cc52bfaca10 in mi_handle_session () from /usr/local/lib/libmilter.so.4.0 #17 0x1cc52bfab7d9 in mi_thread_handle_wrapper () from /usr/local/lib/libmilter.so.4.0 #18 0x1cc5a247d90e in _rthread_start (v=Variable "v" is not available. ) at /usr/src/lib/librthread/rthread.c:145 #19 0x1cc53e33649b in __tfork_thread () at /usr/src/lib/libc/arch/amd64/sys/tfork_thread.S:75 #20 0x in ?? () -- Jeremie Le Hen j...@freebsd.org
