hello Bernhard
there is a lot of files cause it is a syslog for thousands of servers.
between 100k and one million files a day.
coredumpctl gdb 754
PID: 754 (logrotate)
UID: 0 (root)
GID: 0 (root)
Signal: 11 (SEGV)
Timestamp: Sun 2019-01-20 11:24:28 CET (1 day 21h ago)
Command Line: /usr/sbin/logrotate /etc/logrotate.conf
Executable: /usr/sbin/logrotate
Control Group: /system.slice/cron.service
Unit: cron.service
Slice: system.slice
Boot ID: a57707f859fe4471ae781dd31d2b75f7
Machine ID: 230c8c9b6d3840749a45bcf6e73d8a82
Hostname: syslog
Storage:
/var/lib/systemd/coredump/core.logrotate.0.a57707f859fe4471ae781dd31d2b75f7.754.1547979868000000000000.lz4
Message: Process 754 (logrotate) of user 0 dumped core.
Stack trace of thread 754:
#0 0x000055e3239db88a rotateLogSet (logrotate)
#1 0x000055e3239d298d main (logrotate)
#2 0x00007faf357fb2e1 __libc_start_main (libc.so.6)
#3 0x000055e3239d312a _start (logrotate)
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/logrotate...Reading symbols from
/usr/lib/debug/.build-id/4b/a3d893d18935ef292da47c51a97214648caf82.debug...done.
done.
[New LWP 754]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/logrotate /etc/logrotate.conf'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000055e3239db88a in rotateLogSet (log=0x55e323ec4ca0, force=0) at
logrotate.c:1880
1880 logrotate.c: Aucun fichier ou dossier de ce type.
(gdb) print log->numFiles
$1 = 2122453
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 64042
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 63536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 64042
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
dmesg:
[149810.945656] logrotate[1866]: segfault at 7ffc2bbfa628 ip 0000557cef48d88a
sp 00007ffc2bbfa630 error 6 in logrotate[557cef481000+11000]
[233355.909668] logrotate[3273]: segfault at 7fff217853f8 ip 000055bdb39b788a
sp 00007fff21785400 error 6 in logrotate[55bdb39ab000+11000]
[317017.469337] logrotate[4716]: segfault at 7ffc6f3da148 ip 0000561ebf78588a
sp 00007ffc6f3da150 error 6 in logrotate[561ebf779000+11000]
[408729.858051] logrotate[6092]: segfault at 7ffccfec9118 ip 000055ac7751688a
sp 00007ffccfec9120 error 6 in logrotate[55ac7750a000+11000]
[518570.196302] logrotate[8144]: segfault at 7fffaf82c258 ip 00005569827a888a
sp 00007fffaf82c260 error 6 in logrotate[55698279c000+11000]
[1714319.144568] logrotate[127020]: segfault at 7ffe52040b18 ip
0000559fe5b9d88a sp 00007ffe52040b20 error 6 in logrotate[559fe5b91000+11000]
[1877421.532528] logrotate[129941]: segfault at 7fff62c00088 ip
00005586080ba88a sp 00007fff62c00090 error 6 in logrotate[5586080ae000+11000]
[1963749.763571] logrotate[754]: segfault at 7fff97c94a28 ip 000055e3239db88a
sp 00007fff97c94a30 error 6 in logrotate[55e3239cf000+11000]
[2051282.835131] logrotate[2232]: segfault at 7ffec87ae508 ip 0000562ea593b88a
sp 00007ffec87ae510 error 6 in logrotate[562ea592f000+11000]
best regards
thanks
----- Mail original -----
De: "Bernhard Übelacker" <[email protected]>
À: [email protected]
Cc: [email protected]
Envoyé: Lundi 21 Janvier 2019 21:57:16
Objet: Bug#918106: logrotate: segfaults in rotateLogSet
Hello Marc,
> but i don't see much more withour bt full, do i understand correctly ?
I just wanted to see the line from dmesg, where we can see which
address caused the error. And wanted to see all for the same crash.
And I had not realized that the callc instruction really have to
write its return address onto the stack where the ret instruction
can take it later.
[1963749.763571] logrotate[754]: segfault at 7fff97c94a28 ip
000055e3239db88a sp 00007fff97c94a30 error 6 in logrotate[55e3239cf000+11000]
frame #0: rsp 0x7fff97c94a30 0x7fff97c94a30
frame #1: rsp 0x7fff984ad620 0x7fff984ad620
So the line from dmesg shows we want to write to address 7fff97c94a28
that is really near the stack pointer $rsp in the frame #0 (rotateLogSet).
0x7fff984ad620 - 0x7fff97c94a30 = 0x818BF0 == 0n8489968
$ ulimit -a
...
stack size (kbytes, -s) 8192
The difference between the stack pointers in rotateLogSet and main
is around 8291 kb. And what I could read in the net the default
maximum stack size is 8192 kb. Therefore we are trying to write
beyond our stack limit?
Ok, one source of stack exhaustion is recursion, but there
is no evidence for this as we have just a few frames.
The other source that I know of are large function local
arrays and that led me to line 1874:
1870 int rotateLogSet(struct logInfo *log, int force)
1871 {
1872 int i, j;
1873 int hasErrors = 0;
1874 int logHasErrors[log->numFiles];
1875 int numRotated = 0;
The array size depends dynamically on log->numFiles.
But it would take a lot of files to get to that limit ...
(but you wrote that you rotate a lot data)
So could you please provide the output of following command again:
coredumpctl gdb 754
print log->numFiles
Kind regards,
Bernhard
