Re: problems with syslogd [solved]
On Thu, Jun 10, 2010 at 8:54 AM, Oron Peled wrote: > > On Wednesday, 9 בJune 2010 15:14:44 Amit Aronovitch wrote: > > Recently I stopped getting any messages in /var/log/messages (and probably > > ... > > 1) /etc/syslogd.conf is debian's standard, seems to support > > /var/log/messages (as ever): > > > > *.=info;*.=notice;*.=warn;\ > > auth,authpriv.none;\ > > cron,daemon.none;\ > > mail,news.none -/var/log/messages > > Maybe the filename is separated from the selector by spaces instead > of tabs? It is important (at least in legacy syslogd), because > spaces are allow withing the selector expression. > Thanks, that nailed it! Well, not exactly, but close enough to figure it out: The separator was tab all right, but the problem was with the continuation lines. They start with tab. When I delete the initial tab, everything works fine. Seems like the escaped EOL does not skip initial whitespace. I'll report the config bug to Debian. Also the manpage is definitely misleading: syslog.conf(5) says: "Every rule consists of two fields, a selector field and an action field. These two fields are separated by one or more spaces or tabs." Also credit to shimi for mentioning the -f flag of strace (saved me some manpage searching time) - from the trace I saw that it opens the files in the order they appear in the conf file, but it skips "debug" and "messages" - which are exactly the ones that have a continuation line. and thanks all the rest who responded. All tips were good ideas. Amit ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: problems with syslogd
On Wednesday, 9 בJune 2010 15:14:44 Amit Aronovitch wrote: > Recently I stopped getting any messages in /var/log/messages (and probably > ... > 1) /etc/syslogd.conf is debian's standard, seems to support > /var/log/messages (as ever): > > *.=info;*.=notice;*.=warn;\ > auth,authpriv.none;\ > cron,daemon.none;\ > mail,news.none-/var/log/messages Maybe the filename is separated from the selector by spaces instead of tabs? It is important (at least in legacy syslogd), because spaces are allow withing the selector expression. -- Oron Peled Voice: +972-4-8228492 o...@actcom.co.il http://users.actcom.co.il/~oron "Beware of bugs in the above code; I have only proved it correct, not tried it." -- Donald E. Knuth ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: problems with syslogd
2010/6/9 Amit Aronovitch > Hi, > > Recently I stopped getting any messages in /var/log/messages (and probably > some other files as well). Basic tests I could think of all check out OK > (see below). Any ideas what I should check next? > Using sysklogd+klogd 1.5 on Debian (unstable). > > Permission issues? chattr +i / chattr +u ? SELinux / similar ? Of course that your idea to run strace -eopen on daemon startup to see what's going on would be best... HTH, -- Shimi ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: problems with syslogd
On Wed, Jun 9, 2010 at 3:27 PM, linux.il wrote: > > > > > 5) Google found some similar problem reports, but they all turned out to be > > either filesize overflow (have plenty of place on the /var/ partition btw), > > or crashed daemon. > > > may be your /var is out of inodes? nope, I think: penguin:~# df -i /var/ Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda7 6553600 565332 5988268 9% / ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: problems with syslogd
On Wed, Jun 9, 2010 at 3:26 PM, Valery Reznic wrote:
> Connect to syslogd with strace:
>
> strace -p syslogd_pid
>
> And then "provoke" message that should go to /var/log/messages
> strace will show you what syslogd do.
> May be it will reveal cause of the problem.
>
Does not help much. After the "select", it just stats /etc/localtime, and
then writes the output to one of the open fd's
(see my prev mail - the process does not open "messages" at all). For
example:
select(1, [0], NULL, NULL, NULL)= 1 (in [0])
recvfrom(0, "<150>Jun 9 17:33:27 pppd[3998]:"..., 1022, 0, NULL, NULL) = 79
rt_sigprocmask(SIG_BLOCK, [HUP ALRM], NULL, 8) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=803, ...}) = 0
writev(2, [{"Jun 9 17:33:27", 15}, {" ", 1}, {"penguin", 7}, {" ", 1},
{"pppd[3998]: Sent 29064396 bytes,"..., 58}, {"\n", 1}], 6) = 83
rt_sigprocmask(SIG_UNBLOCK, [HUP ALRM], NULL, 8) = 0
select(1, [0], NULL, NULL, NULL)= 1 (in [0])
Now, <150>, according to syslog.h is "local2.info", which should go to
messages because of the following line in /etc/syslog.conf:
*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -/var/log/messages
Seems like I should trace it at daemon startup stage to see if it fails
opening the file...
thanks
___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: problems with syslogd
Connect to syslogd with strace:
strace -p syslogd_pid
And then "provoke" message that should go to /var/log/messages strace will show
you what syslogd do.May be it will reveal cause of the problem.
Valery
--- On Wed, 6/9/10, Amit Aronovitch wrote:
From: Amit Aronovitch
Subject: problems with syslogd
To: "Linux-IL"
Date: Wednesday, June 9, 2010, 3:14 PM
Hi,
Recently I stopped getting any messages in /var/log/messages (and probably some
other files as well). Basic tests I could think of all check out OK (see
below). Any ideas what I should check next?
Using sysklogd+klogd 1.5 on Debian (unstable).
1) /etc/syslogd.conf is debian's standard, seems to support /var/log/messages
(as ever):
(some comment lines truncated)
--->
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
user.* -/var/log/user.log
#
mail.info -/var/log/mail.info
mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err
#
news.crit /var/log/news/news.crit
news.err /var/log/news/news.err
news.notice -/var/log/news/news.notice
#
*.=debug;\
auth,authpriv.none;\
news.none;mail.none -/var/log/debug
*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -/var/log/messages
#
*.emerg *
#
#daemon,mail.*;\
# news.=crit;news.=err;news.=notice;\
# *.=debug;*.=info;\
# *.=notice;*.=warn /dev/tty8
#
daemon.*;mail.*;\
news.err;\
*.=debug;*.=info;\
*.=notice;*.=warn |/dev/xconsole
<--
2) syslogd is running, and has some log files open (but not /var/log/messages
and friends!)
~# ls -al /proc/`ps -C syslogd -o pid=`/fd
total 0
dr-x-- 2 root root 0 Jun 9 14:20 .
dr-xr-xr-x 7 root root 0 Jun 9 14:19 ..
lrwx-- 1 root root 64 Jun 9 14:20 0 -> socket:[1007451]
l-wx-- 1 root root 64 Jun 9 14:20 1 -> /var/log/auth.log
l-wx-- 1 root root 64 Jun 9 14:20 10 -> /var/log/mail.err
l-wx-- 1 root root 64 Jun 9 14:20 11 -> /var/log/news/news.crit
l-wx-- 1 root root 64 Jun 9 14:20 12 -> /var/log/news/news.err
l-wx-- 1 root root 64 Jun 9 14:20 13 -> /var/log/news/news.notice
l-wx-- 1 root root 64 Jun 9 14:20 2 -> /var/log/syslog
l-wx-- 1 root root 64 Jun 9 14:20 3 -> /var/log/daemon.log
l-wx-- 1 root root 64 Jun 9 14:20 4 -> /var/log/kern.log
l-wx-- 1 root root 64 Jun 9 14:20 5 -> /var/log/lpr.log
l-wx-- 1 root root 64 Jun 9 14:20 6 -> /var/log/mail.log
l-wx-- 1 root root 64 Jun 9 14:20 7 -> /var/log/user.log
l-wx-- 1 root root 64 Jun 9 14:20 8 -> /var/log/mail.info
l-wx-- 1 root root 64 Jun 9 14:20 9 -> /var/log/mail.warn
3) log files exist, and seem to have the same permissions as the working ones:
~$ ls -alt `cat /etc/syslog.conf | awk '(substr($1,1,1)!="#" && $2!="")
{sub("-","",$2); if ($2 ~ /^\/var/) print $2}'`
-rw-r- 1 root adm 8025 Jun 9 15:02 /var/log/syslog
-rw-r- 1 root adm 87932 Jun 9 15:02 /var/log/auth.log
-rw-r- 1 root adm 161406 Jun 9 14:19 /var/log/kern.log
-rw-r- 1 root adm 62494 Jun 9 14:00 /var/log/daemon.log
-rw-r- 1 root adm 23295 Jun 9 08:07 /var/log/user.log
-rw-r- 1 root adm 0 Jun 3 08:19 /var/log/debug
-rw-r- 1 root adm 0 Jun 3 08:19 /var/log/messages
-rw-r- 1 root adm 0 Apr 18 06:57 /var/log/mail.info
-rw-r- 1 root adm 0 Apr 18 06:57 /var/log/mail.log
-rw-r- 1 root adm 0 Apr 18 06:57 /var/log/mail.err
-rw-r- 1 root adm 0 Apr 18 06:57 /var/log/mail.warn
-rw-r- 1 root adm 0 Nov 25 2007 /var/log/lpr.log
-rw-r- 1 root adm 0 Feb 20 2005 /var/log/news/news.crit
-rw-r- 1 root adm 0 Feb 20 2005 /var/log/news/news.err
-rw-r- 1 root adm 0 Feb 20 2005 /var/log/news/news.notice
4) Removing and reinstalling the sysklogd package did not help.
5) Google found some similar problem reports, but they all turned out to be
either filesize overflow (have plenty of place on the /var/ partition btw), or
crashed daemon.
What next?
thanks,
AA
-Inline Attachment Follows-
___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: problems with syslogd
> > 5) Google found some similar problem reports, but they all turned out to be > either filesize overflow (have plenty of place on the /var/ partition btw), > or crashed daemon. > may be your /var is out of inodes? ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
