On Wed, 12 Jun 2024 07:19:02 -0300 David Bremner <da...@tethera.net> wrote:
David Bremner <da...@tethera.net> writes:
>
> Attempt #3
>
> swaks -t brem...@debian.org --pipe 'valgrind /usr/lib/sendmail -bs'
>
> This also runs without errors, so I'm out of ideas for the moment.
Attempt #4:
Rebuild with asan
Hello David, hello Axel,
I tried if I can reproduce it and if rr-debugger might be able to record it.
And I received following replay session of the crashing process (PID 3025:
- It forks another process nullmailer-queue (in this example PID 3026)
- It calls waitpid on the PID 3026, but this returns -1 and errno=10 (seems to
be ECHILD)
for some unknown reason.
- Therefore tries to write a error message, but the content and the pointer to
cli_program
seem to get mixed up, therefore the strlen crashes.
Especially the third point is puzzling, I could not yet see why this
pointer-content-mixup happens.
Kind regards,
Bernhard
root@debian:~# rr replay --debugger-option=-q -p 3025 -g 2000 bash-1
root@debian:~# swaks -t a...@debian.org --pipe 'sendmail -bs'
=== Trying pipe to sendmail -bs...
=== Connected to sendmail -bs.
Reading symbols from
/root/.local/share/rr/bash-1/mmap_pack_11_nullmailer-smtpd...
Reading symbols from
/usr/lib/debug/.build-id/42/8c5f859ee211c1dfa8accdc66572493c471db1.debug...
Really redefine built-in command "restart"? (y or n) [answered Y; input not
from terminal]
Really redefine built-in command "jump"? (y or n) [answered Y; input not from
terminal]
Remote debugging using 127.0.0.1:7992
--------------------------------------------------
---> Reached target process 3025 at event 2000.
--------------------------------------------------
Reading symbols from /usr/bin/../lib/rr/librrpreload.so...
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...
Reading symbols from
/usr/lib/debug/.build-id/bc/40b1b7b2a76bc40d2372f4036be41bef33ef14.debug...
Reading symbols from /lib/x86_64-linux-gnu/libstdc++.so.6...
(No debugging symbols found in /lib/x86_64-linux-gnu/libstdc++.so.6)
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...
Reading symbols from
/usr/lib/debug/.build-id/2e/01923fea4ad9f7fa50fe24e0f3385a45a6cd1c.debug...
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...
Reading symbols from
/usr/lib/debug/.build-id/6c/771bfaca294a4e3d85ada43b358bf6b49dcd2a.debug...
Reading symbols from /lib64/ld-linux-x86-64.so.2...
Reading symbols from
/usr/lib/debug/.build-id/a9/700083811ae36d1017fe16ebe5657d59cdda0a.debug...
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...
(No debugging symbols found in /lib/x86_64-linux-gnu/libgcc_s.so.1)
BFD: warning: system-supplied DSO at 0x6fffd000 has a section extending past
end of file
0x00007f0527d7f8b7 in __GI_mprotect () at ../sysdeps/unix/syscall-template.S:117
117 ../sysdeps/unix/syscall-template.S: Datei oder Verzeichnis nicht
gefunden.
(rr) set width 0
(rr) set pagination off
(rr) directory
/home/benutzer/source/nullmailer/orig/nullmailer-2.2+10~g7ed88a0/lib
Source directories searched:
/home/benutzer/source/nullmailer/orig/nullmailer-2.2+10~g7ed88a0/lib:$cdir:$cwd
(rr) directory
/home/benutzer/source/nullmailer/orig/nullmailer-2.2+10~g7ed88a0/src
Source directories searched:
/home/benutzer/source/nullmailer/orig/nullmailer-2.2+10~g7ed88a0/src:/home/benutzer/source/nullmailer/orig/nullmailer-2.2+10~g7ed88a0/lib:$cdir:$cwd
(rr) display/i $pc
1: x/i $pc
=> 0x7f0527d7f8b7 <__GI_mprotect+7>: cmp $0xfffffffffffff001,%rax
(rr) b fork
Breakpoint 1 at 0x7f05278f5560: file ./posix/fork.c, line 41.
(rr) cont
Continuing.
<- 220 nullmailer-smtpd ready
-> EHLO debian
<- 250 2.3.0 OK
-> MAIL FROM:<root@debian>
<- 250 2.1.0 Sender accepted
-> RCPT TO:<a...@debian.org>
<- 250 2.1.5 Recipient accepted
-> DATA
Breakpoint 1, __libc_fork () at ./posix/fork.c:41
41 ./posix/fork.c: Datei oder Verzeichnis nicht gefunden.
1: x/i $pc
=> 0x7f05278f5560 <__libc_fork>: push %r14
(rr) finish
Run till exit from #0 __libc_fork () at ./posix/fork.c:41
0x00005639bd9132d5 in fork_exec::start (this=this@entry=0x7ffc4099cd00,
args=args@entry=0x7ffc4099cca0, redirn=redirn@entry=3,
redirs=redirs@entry=0x7ffc4099cc90) at ./lib/forkexec.cc:67
67 if ((pid = fork()) < 0)
1: x/i $pc
=> 0x5639bd9132d5 <_ZN9fork_exec5startEPPKciPi+501>: mov -0x58(%rbp),%rsi
Value returned is $1 = 3026
(rr) next
69 if (pid == 0) {
1: x/i $pc
=> 0x5639bd9132e3 <_ZN9fork_exec5startEPPKciPi+515>: jne 0x5639bd913306
<_ZN9fork_exec5startEPPKciPi+550>
(rr) print pid
$2 = 3026
(rr) b fork_exec::wait
Breakpoint 2 at 0x5639bd912ea0: file ./lib/forkexec.cc, line 121.
(rr) cont
Continuing.
<- 354 End your message with a period on a line by itself
-> Date: Fri, 21 Jun 2024 00:10:45 +0200
-> To: a...@debian.org
-> From: root@debian
-> Subject: test Fri, 21 Jun 2024 00:10:45 +0200
-> Message-Id: <20240621001045.003024@debian>
-> X-Mailer: swaks v20240103.0 jetmore.org/john/code/swaks/
->
-> This is a test mailing
->
->
-> .
Breakpoint 2, fork_exec::wait (this=this@entry=0x7ffc4099cd00) at
./lib/forkexec.cc:121
121 {
1: x/i $pc
=> 0x5639bd912ea0 <_ZN9fork_exec4waitEv>: endbr64
(rr) next
122 if (pid > 0) {
1: x/i $pc
=> 0x5639bd912ec1 <_ZN9fork_exec4waitEv+33>: test %edi,%edi
(rr)
123 int status = wait_status();
1: x/i $pc
=> 0x5639bd912ef0 <_ZN9fork_exec4waitEv+80>: xor %edx,%edx
(rr) step
fork_exec::wait_status (this=0x7ffc4099cd00) at ./lib/forkexec.cc:112
112 if (waitpid(pid, &status, 0) == pid) {
1: x/i $pc
=> 0x5639bd912ef0 <_ZN9fork_exec4waitEv+80>: xor %edx,%edx
(rr) stepi
0x00005639bd912ef2 112 if (waitpid(pid, &status, 0) == pid) {
1: x/i $pc
=> 0x5639bd912ef2 <_ZN9fork_exec4waitEv+82>: lea 0x4(%rsp),%rsi
(rr)
0x00005639bd912ef7 112 if (waitpid(pid, &status, 0) == pid) {
1: x/i $pc
=> 0x5639bd912ef7 <_ZN9fork_exec4waitEv+87>: call 0x5639bd9110f0
<waitpid@plt>
(rr)
0x00005639bd9110f0 in waitpid@plt ()
1: x/i $pc
=> 0x5639bd9110f0 <waitpid@plt>: jmp *0x6e4a(%rip) # 0x5639bd917f40
<wait...@got.plt>
(rr)
__GI___waitpid (pid=3026, stat_loc=0x7ffc4099ccb4, options=0) at
./posix/waitpid.c:38
38 ./posix/waitpid.c: Datei oder Verzeichnis nicht gefunden.
1: x/i $pc
=> 0x7f05279109e0 <__GI___waitpid>: xor %ecx,%ecx
(rr) finish
Run till exit from #0 __GI___waitpid (pid=3026, stat_loc=0x7ffc4099ccb4,
options=0) at ./posix/waitpid.c:38
0x00005639bd912efc in fork_exec::wait_status (this=0x7ffc4099cd00) at
./lib/forkexec.cc:112
112 if (waitpid(pid, &status, 0) == pid) {
1: x/i $pc
=> 0x5639bd912efc <_ZN9fork_exec4waitEv+92>: cmp (%rbx),%eax
Value returned is $3 = -1
(rr) print status
$4 = 0
(rr) print errno
$5 = 10
(rr) stepi
0x00005639bd912efe 112 if (waitpid(pid, &status, 0) == pid) {
1: x/i $pc
=> 0x5639bd912efe <_ZN9fork_exec4waitEv+94>: jne 0x5639bd913058
<_ZN9fork_exec4waitEv+440>
(rr)
0x00005639bd913058 in fork_exec::wait (this=this@entry=0x7ffc4099cd00) at
./lib/forkexec.cc:134
134 FAIL(name << " crashed or was killed");
1: x/i $pc
=> 0x5639bd913058 <_ZN9fork_exec4waitEv+440>: mov 0x20e1(%rip),%rbp #
0x5639bd915140 <cli_program>
(rr) x/2xg $rip + 0x20e1
0x5639bd915139: 0x6e00000000000000 0x656c69616d6c6c75
(rr) print cli_program
$6 = "nullmailer-smtpd"
(rr) stepi
fdobuf::operator<< (str=<optimized out>, this=<optimized out>) at
./fdbuf/fdobuf.h:59
59 write(str, strlen(str));
1: x/i $pc
=> 0x5639bd91305f <_ZN9fork_exec4waitEv+447>: lea 0x201f(%rip),%r12
# 0x5639bd915085
(rr)
0x00005639bd913066 in fdobuf::operator<< (str=0x6c69616d6c6c756e <error: Cannot access
memory at address 0x6c69616d6c6c756e>, this=<optimized out>) at ./fdbuf/fdobuf.h:59
59 write(str, strlen(str));
1: x/i $pc
=> 0x5639bd913066 <_ZN9fork_exec4waitEv+454>: mov %rbp,%rdi
(rr)
0x00005639bd913069 59 write(str, strlen(str));
1: x/i $pc
=> 0x5639bd913069 <_ZN9fork_exec4waitEv+457>: call 0x5639bd9110a0
<strlen@plt>
(rr)
0x00005639bd9110a0 in strlen@plt ()
1: x/i $pc
=> 0x5639bd9110a0 <strlen@plt>: jmp *0x6e72(%rip) # 0x5639bd917f18
<str...@got.plt>
(rr)
__strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:65
65 ../sysdeps/x86_64/multiarch/strlen-avx2.S: Datei oder Verzeichnis nicht
gefunden.
1: x/i $pc
=> 0x7f0527973dc0 <__strlen_avx2>: mov %edi,%eax
(rr) finish
Run till exit from #0 __strlen_avx2 () at
../sysdeps/x86_64/multiarch/strlen-avx2.S:65
Program received signal SIGSEGV, Segmentation fault.
__strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76
76 in ../sysdeps/x86_64/multiarch/strlen-avx2.S
1: x/i $pc
=> 0x7f0527973dd9 <__strlen_avx2+25>: vpcmpeqb (%rdi),%ymm0,%ymm1
(rr) bt
#0 __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76
#1 0x00005639bd91306e in fdobuf::operator<< (str=0x6c69616d6c6c756e <error: Cannot
access memory at address 0x6c69616d6c6c756e>, this=<optimized out>) at
./fdbuf/fdobuf.h:59
#2 fork_exec::wait (this=this@entry=0x7ffc4099cd00) at ./lib/forkexec.cc:125
#3 0x00005639bd91201f in DATA (param=...) at ./src/smtpd.cc:159
#4 DATA (param=...) at ./src/smtpd.cc:127
#5 0x00005639bd91144f in dispatch () at ./src/smtpd.cc:252
#6 main () at ./src/smtpd.cc:263
(rr) list 34
34
35 #define ERR(MSG) do{ ferr << cli_program << ": " << MSG << ": " <<
strerror(errno) << endl; } while(0)
36 #define FAIL(MSG) do{ ERR(MSG); return false; }while(0)
37
(rr) list fork_exec::wait
119
120 bool fork_exec::wait()
121 {
122 if (pid > 0) {
123 int status = wait_status();
124 if (status < 0)
125 FAIL("Error catching the return value from " << name);
126 if (WIFEXITED(status)) {
127 status = WEXITSTATUS(status);
128 if (status) {
129 ferr << cli_program << ": " << name << " failed: " << status <<
endl;
130 return false;
131 }
132 }
133 else
134 FAIL(name << " crashed or was killed");
135 }
136 return true;
137 }
(rr)
# Trixie/testing amd64 qemu VM 2024-06-20
apt install systemd-coredump mc rr gdb swaks nullmailer nullmailer-dbgsym
apt build-dep nullmailer
mkdir /home/benutzer/source/nullmailer/orig -p
cd /home/benutzer/source/nullmailer/orig
apt source nullmailer
swaks -t a...@debian.org --pipe 'sendmail -bs'
Jun 21 00:02:23 debian nullmailer-send[2700]: Trigger pulled.
Jun 21 00:02:23 debian nullmailer-send[2700]: Rescanning queue.
Jun 21 00:02:23 debian nullmailer-send[2700]: Starting delivery, 1 message(s)
in queue.
Jun 21 00:02:23 debian nullmailer-send[2700]: Starting delivery: host: mail
protocol: smtp file: 1718920943.2751
Jun 21 00:02:23 debian nullmailer-send[2700]: From: <root@debian> to:
<a...@debian.org>
Jun 21 00:02:23 debian nullmailer-send[2700]: Message-Id:
<20240621000223.002749@debian>
Jun 21 00:02:23 debian kernel: traps: nullmailer-smtp[2750] general protection
fault ip:7f56ee573dd9 sp:7ffc8ab12ef8 error:0 in libc.so.6[7f56ee441000+157000]
Jun 21 00:02:23 debian systemd-coredump[2753]: Process 2750 (nullmailer-smtp)
of user 0 terminated abnormally with signal 11/SEGV, processing...
Jun 21 00:02:23 debian nullmailer-send[2752]: smtp: Failed: Connect failed
Jun 21 00:02:23 debian nullmailer-send[2700]: Sending failed: Host not found
Jun 21 00:02:23 debian nullmailer-send[2700]: Delivery complete, 1 message(s)
remain.
Jun 21 00:02:23 debian systemd[1]: Started systemd-coredump@1-2753-0.service -
Process Core Dump (PID 2753/UID 0).
Jun 21 00:02:23 debian systemd-coredump[2754]: Resource limits disable core
dumping for process 2750 (nullmailer-smtp).
Jun 21 00:02:23 debian systemd-coredump[2754]: [🡕] Process 2750
(nullmailer-smtp) of user 0 terminated abnormally without generating a coredump.
Jun 21 00:02:23 debian systemd[1]: systemd-coredump@1-2753-0.service:
Deactivated successfully.
root@debian:~# coredumpctl list
TIME PID UID GID SIG COREFILE EXE
SIZE
Fri 2024-06-21 00:02:23 CEST 2750 0 0 SIGSEGV none
/usr/bin/nullmailer-smtpd -
root@debian:~# coredumpctl gdb --debugger-argument=-q 2750
PID: 2750 (nullmailer-smtp)
UID: 0 (root)
GID: 0 (root)
Signal: 11 (SEGV)
Timestamp: Fri 2024-06-21 00:02:23 CEST (6min ago)
Command Line: /usr/bin/nullmailer-smtpd
Executable: /usr/bin/nullmailer-smtpd
Control Group: /user.slice/user-1000.slice/session-3.scope
Unit: session-3.scope
Slice: user-1000.slice
Session: 3
Owner UID: 1000 (benutzer)
Boot ID: 7d564518406445b188aaedbfe2bae6f0
Machine ID: 16e4d7437c19482b8c85581d3feaba09
Hostname: debian
Storage: none
Message: Process 2750 (nullmailer-smtp) of user 0 terminated abnormally
without generating a coredump.
Coredump entry has no core attached (neither internally in the journal nor
externally on disk).
root@debian:~#
root@debian:~# rr record bash
rr: Saving execution to trace directory `/root/.local/share/rr/bash-1'.
root@debian:~# swaks -t a...@debian.org --pipe 'sendmail -bs'
=== Trying pipe to sendmail -bs...
=== Connected to sendmail -bs.
<- 220 nullmailer-smtpd ready
-> EHLO debian
<- 250 2.3.0 OK
-> MAIL FROM:<root@debian>
<- 250 2.1.0 Sender accepted
-> RCPT TO:<a...@debian.org>
<- 250 2.1.5 Recipient accepted
-> DATA
<- 354 End your message with a period on a line by itself
-> Date: Fri, 21 Jun 2024 00:10:45 +0200
-> To: a...@debian.org
-> From: root@debian
-> Subject: test Fri, 21 Jun 2024 00:10:45 +0200
-> Message-Id: <20240621001045.003024@debian>
-> X-Mailer: swaks v20240103.0 jetmore.org/john/code/swaks/
->
-> This is a test mailing
->
->
-> .
*** Child process closed connection unexpectedly.
root@debian:~# exit
exit
root@debian:~#
root@debian:~# rr pack
rr: Packed trace directory `/root/.local/share/rr/bash-1'.
root@debian:~# rr ps
PID PPID EXIT CMD
3021 -- 6 bash
3024 3021 6 swaks -t a...@debian.org --pipe sendmail -bs
3025 3024 -11 sendmail -bs
3026 3025 0 /usr/sbin/nullmailer-queue
root@debian:~#
rr replay --debugger-option=-q -p 3025 -g 2000 bash-1
set width 0
set pagination off
directory /home/benutzer/source/nullmailer/orig/nullmailer-2.2+10~g7ed88a0/lib
directory /home/benutzer/source/nullmailer/orig/nullmailer-2.2+10~g7ed88a0/src
display/i $pc
cont
root@debian:~# rr replay --debugger-option=-q -p 3025 -g 2000 bash-1
root@debian:~# swaks -t a...@debian.org --pipe 'sendmail -bs'
=== Trying pipe to sendmail -bs...
=== Connected to sendmail -bs.
Reading symbols from
/root/.local/share/rr/bash-1/mmap_pack_11_nullmailer-smtpd...
Reading symbols from
/usr/lib/debug/.build-id/42/8c5f859ee211c1dfa8accdc66572493c471db1.debug...
Really redefine built-in command "restart"? (y or n) [answered Y; input not
from terminal]
Really redefine built-in command "jump"? (y or n) [answered Y; input not from
terminal]
Remote debugging using 127.0.0.1:4589
--------------------------------------------------
---> Reached target process 3025 at event 2000.
--------------------------------------------------
Reading symbols from /usr/bin/../lib/rr/librrpreload.so...
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...
Reading symbols from
/usr/lib/debug/.build-id/bc/40b1b7b2a76bc40d2372f4036be41bef33ef14.debug...
Reading symbols from /lib/x86_64-linux-gnu/libstdc++.so.6...
(No debugging symbols found in /lib/x86_64-linux-gnu/libstdc++.so.6)
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...
Reading symbols from
/usr/lib/debug/.build-id/2e/01923fea4ad9f7fa50fe24e0f3385a45a6cd1c.debug...
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...
Reading symbols from
/usr/lib/debug/.build-id/6c/771bfaca294a4e3d85ada43b358bf6b49dcd2a.debug...
Reading symbols from /lib64/ld-linux-x86-64.so.2...
Reading symbols from
/usr/lib/debug/.build-id/a9/700083811ae36d1017fe16ebe5657d59cdda0a.debug...
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...
(No debugging symbols found in /lib/x86_64-linux-gnu/libgcc_s.so.1)
BFD: warning: system-supplied DSO at 0x6fffd000 has a section extending past
end of file
0x00007f0527d7f8b7 in __GI_mprotect () at ../sysdeps/unix/syscall-template.S:117
117 ../sysdeps/unix/syscall-template.S: Datei oder Verzeichnis nicht
gefunden.
(rr) set width 0
(rr) set pagination off
(rr) directory
/home/benutzer/source/nullmailer/orig/nullmailer-2.2+10~g7ed88a0/lib
Source directories searched:
/home/benutzer/source/nullmailer/orig/nullmailer-2.2+10~g7ed88a0/lib:$cdir:$cwd
(rr) cont
Continuing.
<- 220 nullmailer-smtpd ready
-> EHLO debian
<- 250 2.3.0 OK
-> MAIL FROM:<root@debian>
<- 250 2.1.0 Sender accepted
-> RCPT TO:<a...@debian.org>
<- 250 2.1.5 Recipient accepted
-> DATA
<- 354 End your message with a period on a line by itself
-> Date: Fri, 21 Jun 2024 00:10:45 +0200
-> To: a...@debian.org
-> From: root@debian
-> Subject: test Fri, 21 Jun 2024 00:10:45 +0200
-> Message-Id: <20240621001045.003024@debian>
-> X-Mailer: swaks v20240103.0 jetmore.org/john/code/swaks/
->
-> This is a test mailing
->
->
-> .
Program received signal SIGSEGV, Segmentation fault.
__strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76
76 ../sysdeps/x86_64/multiarch/strlen-avx2.S: Datei oder Verzeichnis nicht
gefunden.
(rr) bt
#0 __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76
#1 0x00005639bd91306e in fdobuf::operator<< (str=0x6c69616d6c6c756e <error:
Cannot access memory at address 0x6c69616d6c6c756e>, this=<optimized out>) at
./fdbuf/fdobuf.h:59
#2 fork_exec::wait (this=this@entry=0x7ffc4099cd00) at ./lib/forkexec.cc:125
#3 0x00005639bd91201f in DATA (param=...) at ./src/smtpd.cc:159
#4 DATA (param=...) at ./src/smtpd.cc:127
#5 0x00005639bd91144f in dispatch () at ./src/smtpd.cc:252
#6 main () at ./src/smtpd.cc:263
(rr) display/i $pc
1: x/i $pc
=> 0x7f0527973dd9 <__strlen_avx2+25>: vpcmpeqb (%rdi),%ymm0,%ymm1
(rr) print/x $rdi
$1 = 0x6c69616d6c6c756e
(rr) up
#1 0x00005639bd91306e in fdobuf::operator<< (str=0x6c69616d6c6c756e <error:
Cannot access memory at address 0x6c69616d6c6c756e>, this=<optimized out>) at
./fdbuf/fdobuf.h:59
59 write(str, strlen(str));
(rr) print str
$2 = 0x6c69616d6c6c756e <error: Cannot access memory at address
0x6c69616d6c6c756e>
(rr) list
54 bool chown(uid_t, gid_t) const;
55 bool chmod(mode_t) const;
56
57 fdobuf& operator<<(const char* str)
58 {
59 write(str, strlen(str));
60 return *this;
61 }
62 fdobuf& operator<<(char ch)
63 {
(rr) print &str
Address requested for identifier "str" which is in register $rbp
(rr) display/x $rbp
2: /x $rbp = 0x6c69616d6c6c756e
(rr) down
#0 __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76
76 ../sysdeps/x86_64/multiarch/strlen-avx2.S: Datei oder Verzeichnis nicht
gefunden.
(rr) reverse-finish
...
(rr) reverse-finish
...
(rr) reverse-stepi
0x00005639bd913066 59 write(str, strlen(str));
1: x/i $pc
=> 0x5639bd913066 <_ZN9fork_exec4waitEv+454>: mov %rbp,%rdi
2: /x $rbp = 0x6c69616d6c6c756e
(rr)
fdobuf::operator<< (str=<optimized out>, this=<optimized out>) at
./fdbuf/fdobuf.h:59
59 write(str, strlen(str));
1: x/i $pc
=> 0x5639bd91305f <_ZN9fork_exec4waitEv+447>: lea 0x201f(%rip),%r12
# 0x5639bd915085
2: /x $rbp = 0x6c69616d6c6c756e
(rr)
0x00005639bd913058 in fork_exec::wait (this=this@entry=0x7ffc4099cd00) at
./lib/forkexec.cc:134
134 FAIL(name << " crashed or was killed");
1: x/i $pc
=> 0x5639bd913058 <_ZN9fork_exec4waitEv+440>: mov 0x20e1(%rip),%rbp
# 0x5639bd915140 <cli_program>
2: /x $rbp = 0x4
(rr) print name
$3 = 0x5639bd91506d "nullmailer-queue"
(rr) x/2xg $rip + 0x20e1
0x5639bd915139: 0x6e00000000000000 0x656c69616d6c6c75
(rr) x/1xg 0x5639bd915140
0x5639bd915140 <cli_program>: 0x6c69616d6c6c756e
(rr) list 34
34
35 #define ERR(MSG) do{ ferr << cli_program << ": " << MSG << ": " <<
strerror(errno) << endl; } while(0)
36 #define FAIL(MSG) do{ ERR(MSG); return false; }while(0)
37
(rr) list fork_exec::wait
119
120 bool fork_exec::wait()
121 {
122 if (pid > 0) {
123 int status = wait_status();
124 if (status < 0)
125 FAIL("Error catching the return value from " << name);
126 if (WIFEXITED(status)) {
127 status = WEXITSTATUS(status);
128 if (status) {
129 ferr << cli_program << ": " << name << " failed: " << status <<
endl;
130 return false;
131 }
132 }
133 else
134 FAIL(name << " crashed or was killed");
135 }
136 return true;
137 }
(rr) print ferr
$4 = {<fdbuf> = {buf = 0x5639bf5872d0 "", buflength = 0, bufstart = 0, offset =
0, errnum = 0, flags = 0, bufsize = 4096, fd = 2, do_close = false},
_vptr.fdobuf = 0x5639bd917c40 <vtable for fdobuf+16>, bufpos = 0, count = 0}
(rr) print cli_program
$5 = "nullmailer-smtpd"
(rr) print name
$6 = 0x5639bd91506d "nullmailer-queue"
(rr) print errno
$7 = 10
(rr) print pid
$10 = 3026
(rr) print/c 0x6e
$2 = 110 'n'
(rr) print/c 0x75
$3 = 117 'u'
(rr) print/c 0x6c
$4 = 108 'l'
(rr) print/c 0x6c
$5 = 108 'l'
(rr) print/c 0x6d
$6 = 109 'm'
(rr) print/c 0x61
$7 = 97 'a'
(rr) print/c 0x69
$8 = 105 'i'
(rr) print/c 0x6c
$9 = 108 'l'
root@debian:~# rr replay --debugger-option=-q -p 3025 -g 2000 bash-1
root@debian:~# swaks -t a...@debian.org --pipe 'sendmail -bs'
=== Trying pipe to sendmail -bs...
=== Connected to sendmail -bs.
Reading symbols from
/root/.local/share/rr/bash-1/mmap_pack_11_nullmailer-smtpd...
Reading symbols from
/usr/lib/debug/.build-id/42/8c5f859ee211c1dfa8accdc66572493c471db1.debug...
Really redefine built-in command "restart"? (y or n) [answered Y; input not
from terminal]
Really redefine built-in command "jump"? (y or n) [answered Y; input not from
terminal]
Remote debugging using 127.0.0.1:7992
--------------------------------------------------
---> Reached target process 3025 at event 2000.
--------------------------------------------------
Reading symbols from /usr/bin/../lib/rr/librrpreload.so...
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...
Reading symbols from
/usr/lib/debug/.build-id/bc/40b1b7b2a76bc40d2372f4036be41bef33ef14.debug...
Reading symbols from /lib/x86_64-linux-gnu/libstdc++.so.6...
(No debugging symbols found in /lib/x86_64-linux-gnu/libstdc++.so.6)
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...
Reading symbols from
/usr/lib/debug/.build-id/2e/01923fea4ad9f7fa50fe24e0f3385a45a6cd1c.debug...
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...
Reading symbols from
/usr/lib/debug/.build-id/6c/771bfaca294a4e3d85ada43b358bf6b49dcd2a.debug...
Reading symbols from /lib64/ld-linux-x86-64.so.2...
Reading symbols from
/usr/lib/debug/.build-id/a9/700083811ae36d1017fe16ebe5657d59cdda0a.debug...
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...
(No debugging symbols found in /lib/x86_64-linux-gnu/libgcc_s.so.1)
BFD: warning: system-supplied DSO at 0x6fffd000 has a section extending past
end of file
0x00007f0527d7f8b7 in __GI_mprotect () at ../sysdeps/unix/syscall-template.S:117
117 ../sysdeps/unix/syscall-template.S: Datei oder Verzeichnis nicht
gefunden.
(rr) set width 0
(rr) set pagination off
(rr) directory
/home/benutzer/source/nullmailer/orig/nullmailer-2.2+10~g7ed88a0/lib
Source directories searched:
/home/benutzer/source/nullmailer/orig/nullmailer-2.2+10~g7ed88a0/lib:$cdir:$cwd
(rr) directory
/home/benutzer/source/nullmailer/orig/nullmailer-2.2+10~g7ed88a0/src
Source directories searched:
/home/benutzer/source/nullmailer/orig/nullmailer-2.2+10~g7ed88a0/src:/home/benutzer/source/nullmailer/orig/nullmailer-2.2+10~g7ed88a0/lib:$cdir:$cwd
(rr) display/i $pc
1: x/i $pc
=> 0x7f0527d7f8b7 <__GI_mprotect+7>: cmp $0xfffffffffffff001,%rax
(rr) b fork
Breakpoint 1 at 0x7f05278f5560: file ./posix/fork.c, line 41.
(rr) cont
Continuing.
<- 220 nullmailer-smtpd ready
-> EHLO debian
<- 250 2.3.0 OK
-> MAIL FROM:<root@debian>
<- 250 2.1.0 Sender accepted
-> RCPT TO:<a...@debian.org>
<- 250 2.1.5 Recipient accepted
-> DATA
Breakpoint 1, __libc_fork () at ./posix/fork.c:41
41 ./posix/fork.c: Datei oder Verzeichnis nicht gefunden.
1: x/i $pc
=> 0x7f05278f5560 <__libc_fork>: push %r14
(rr) finish
Run till exit from #0 __libc_fork () at ./posix/fork.c:41
0x00005639bd9132d5 in fork_exec::start (this=this@entry=0x7ffc4099cd00,
args=args@entry=0x7ffc4099cca0, redirn=redirn@entry=3,
redirs=redirs@entry=0x7ffc4099cc90) at ./lib/forkexec.cc:67
67 if ((pid = fork()) < 0)
1: x/i $pc
=> 0x5639bd9132d5 <_ZN9fork_exec5startEPPKciPi+501>: mov -0x58(%rbp),%rsi
Value returned is $1 = 3026
(rr) next
69 if (pid == 0) {
1: x/i $pc
=> 0x5639bd9132e3 <_ZN9fork_exec5startEPPKciPi+515>: jne 0x5639bd913306
<_ZN9fork_exec5startEPPKciPi+550>
(rr) print pid
$2 = 3026
(rr) b fork_exec::wait
Breakpoint 2 at 0x5639bd912ea0: file ./lib/forkexec.cc, line 121.
(rr) cont
Continuing.
<- 354 End your message with a period on a line by itself
-> Date: Fri, 21 Jun 2024 00:10:45 +0200
-> To: a...@debian.org
-> From: root@debian
-> Subject: test Fri, 21 Jun 2024 00:10:45 +0200
-> Message-Id: <20240621001045.003024@debian>
-> X-Mailer: swaks v20240103.0 jetmore.org/john/code/swaks/
->
-> This is a test mailing
->
->
-> .
Breakpoint 2, fork_exec::wait (this=this@entry=0x7ffc4099cd00) at
./lib/forkexec.cc:121
121 {
1: x/i $pc
=> 0x5639bd912ea0 <_ZN9fork_exec4waitEv>: endbr64
(rr) next
122 if (pid > 0) {
1: x/i $pc
=> 0x5639bd912ec1 <_ZN9fork_exec4waitEv+33>: test %edi,%edi
(rr)
123 int status = wait_status();
1: x/i $pc
=> 0x5639bd912ef0 <_ZN9fork_exec4waitEv+80>: xor %edx,%edx
(rr) step
fork_exec::wait_status (this=0x7ffc4099cd00) at ./lib/forkexec.cc:112
112 if (waitpid(pid, &status, 0) == pid) {
1: x/i $pc
=> 0x5639bd912ef0 <_ZN9fork_exec4waitEv+80>: xor %edx,%edx
(rr) stepi
0x00005639bd912ef2 112 if (waitpid(pid, &status, 0) == pid) {
1: x/i $pc
=> 0x5639bd912ef2 <_ZN9fork_exec4waitEv+82>: lea 0x4(%rsp),%rsi
(rr)
0x00005639bd912ef7 112 if (waitpid(pid, &status, 0) == pid) {
1: x/i $pc
=> 0x5639bd912ef7 <_ZN9fork_exec4waitEv+87>: call 0x5639bd9110f0
<waitpid@plt>
(rr)
0x00005639bd9110f0 in waitpid@plt ()
1: x/i $pc
=> 0x5639bd9110f0 <waitpid@plt>: jmp *0x6e4a(%rip) #
0x5639bd917f40 <wait...@got.plt>
(rr)
__GI___waitpid (pid=3026, stat_loc=0x7ffc4099ccb4, options=0) at
./posix/waitpid.c:38
38 ./posix/waitpid.c: Datei oder Verzeichnis nicht gefunden.
1: x/i $pc
=> 0x7f05279109e0 <__GI___waitpid>: xor %ecx,%ecx
(rr) finish
Run till exit from #0 __GI___waitpid (pid=3026, stat_loc=0x7ffc4099ccb4,
options=0) at ./posix/waitpid.c:38
0x00005639bd912efc in fork_exec::wait_status (this=0x7ffc4099cd00) at
./lib/forkexec.cc:112
112 if (waitpid(pid, &status, 0) == pid) {
1: x/i $pc
=> 0x5639bd912efc <_ZN9fork_exec4waitEv+92>: cmp (%rbx),%eax
Value returned is $3 = -1
(rr) print status
$4 = 0
(rr) print errno
$5 = 10
(rr) stepi
0x00005639bd912efe 112 if (waitpid(pid, &status, 0) == pid) {
1: x/i $pc
=> 0x5639bd912efe <_ZN9fork_exec4waitEv+94>: jne 0x5639bd913058
<_ZN9fork_exec4waitEv+440>
(rr)
0x00005639bd913058 in fork_exec::wait (this=this@entry=0x7ffc4099cd00) at
./lib/forkexec.cc:134
134 FAIL(name << " crashed or was killed");
1: x/i $pc
=> 0x5639bd913058 <_ZN9fork_exec4waitEv+440>: mov 0x20e1(%rip),%rbp
# 0x5639bd915140 <cli_program>
(rr) x/2xg $rip + 0x20e1
0x5639bd915139: 0x6e00000000000000 0x656c69616d6c6c75
(rr) print cli_program
$6 = "nullmailer-smtpd"
(rr) stepi
fdobuf::operator<< (str=<optimized out>, this=<optimized out>) at
./fdbuf/fdobuf.h:59
59 write(str, strlen(str));
1: x/i $pc
=> 0x5639bd91305f <_ZN9fork_exec4waitEv+447>: lea 0x201f(%rip),%r12
# 0x5639bd915085
(rr)
0x00005639bd913066 in fdobuf::operator<< (str=0x6c69616d6c6c756e <error: Cannot
access memory at address 0x6c69616d6c6c756e>, this=<optimized out>) at
./fdbuf/fdobuf.h:59
59 write(str, strlen(str));
1: x/i $pc
=> 0x5639bd913066 <_ZN9fork_exec4waitEv+454>: mov %rbp,%rdi
(rr)
0x00005639bd913069 59 write(str, strlen(str));
1: x/i $pc
=> 0x5639bd913069 <_ZN9fork_exec4waitEv+457>: call 0x5639bd9110a0
<strlen@plt>
(rr)
0x00005639bd9110a0 in strlen@plt ()
1: x/i $pc
=> 0x5639bd9110a0 <strlen@plt>: jmp *0x6e72(%rip) # 0x5639bd917f18
<str...@got.plt>
(rr)
__strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:65
65 ../sysdeps/x86_64/multiarch/strlen-avx2.S: Datei oder Verzeichnis nicht
gefunden.
1: x/i $pc
=> 0x7f0527973dc0 <__strlen_avx2>: mov %edi,%eax
(rr) finish
Run till exit from #0 __strlen_avx2 () at
../sysdeps/x86_64/multiarch/strlen-avx2.S:65
Program received signal SIGSEGV, Segmentation fault.
__strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76
76 in ../sysdeps/x86_64/multiarch/strlen-avx2.S
1: x/i $pc
=> 0x7f0527973dd9 <__strlen_avx2+25>: vpcmpeqb (%rdi),%ymm0,%ymm1
(rr) bt
#0 __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76
#1 0x00005639bd91306e in fdobuf::operator<< (str=0x6c69616d6c6c756e <error:
Cannot access memory at address 0x6c69616d6c6c756e>, this=<optimized out>) at
./fdbuf/fdobuf.h:59
#2 fork_exec::wait (this=this@entry=0x7ffc4099cd00) at ./lib/forkexec.cc:125
#3 0x00005639bd91201f in DATA (param=...) at ./src/smtpd.cc:159
#4 DATA (param=...) at ./src/smtpd.cc:127
#5 0x00005639bd91144f in dispatch () at ./src/smtpd.cc:252
#6 main () at ./src/smtpd.cc:263
(rr)
