Hi all,
After much battling I've managed to increase the number of file handles
available to rsyslog in Ubuntu 14.04. It was significantly harder than
you might expect, even given that Ubuntu uses upstart, which does
strange stuff with ulimits. So I thought it would be worth sharing the
trick that works with the list. Perhaps someone could even explain to
me *why* it works :-) Or, even better, tell me a better way to do it.
Or both...
Scenario: I wanted to increase the number of available file handles for
syslog, because I'm writing to a large number of files via a large
number of queues.
Short version: you need to put "ulimit -n 16384" (or whatever) inside
/etc/default/rsyslog. The normal way of setting the open file limit for
an upstart job (ie. "limit nofile XXX") does not work, at least for me.
Putting the ulimit call inside the upstart rsyslog.conf doesn't work
either, which is weird (for reasons explained below).
Long version: the default Ubuntu upstart script for starting rsyslog
looks like this:
# rsyslog - system logging daemon
#
# rsyslog is an enhanced multi-threaded replacement for the traditional
# syslog daemon, logging messages from applications
description "system logging daemon"
start on filesystem
stop on runlevel [06]
expect fork
respawn
pre-start script
/lib/init/apparmor-profile-load usr.sbin.rsyslogd
end script
script
. /etc/default/rsyslog
exec rsyslogd $RSYSLOGD_OPTIONS
end script
Now, upstart does its own ulimit handling, and changes to limits.conf
don't affect processes launched by it, so the normal way to increase the
number of file handles is to add a line like this just after the "expect
fork"/"respawn" bit:
limit nofile 16384 16384
Indeed, David Lang suggested doing exactly that a couple of months ago
in a different thread (archive
<http://lists.adiscon.net/pipermail/rsyslog/2015-October/041404.html>).
But, weirdly, it doesn't work, at least for me. Inspecting
/proc/<rsyslog-pid>/limits after adding that line and bouncing rsyslog
showed that the file handle limit was still 1024 soft, 4096 hard. And I
found that my system was not writing to a number of the files it was
meant to; "ls /proc/<rsyslog-pid>/fd/ | wc -l" confirmed that it was
capping out at 1024 files open.
My first guess was that it's something to do with the fact that Ubuntu's
upstart script launches rsyslog via a "script" section. The normal way
to start a process in upstart is simply something like
exec <binary> <options>
...without a "script" block around it. The script block just allows you
to add normal shell script to the upstart config so that you can do
something more complicated. So, I thought, perhaps the "limit nofile"
stuff only works if you're using a simple exec, and for scripts you have
to do stuff manually. I added a
ulimit -n 16384
...inside the script tag. But that didn't work.
The next thing was investigate *why* the default upstart config uses the
script tag. Best guess is that it's there so that the contents of
/etc/default/rsyslog can be read, so that configuration options can be
placed in that file. Looking inside /etc/default/rsyslog showed that
the default file simply contained this:
# Options for rsyslogd
# -x disables DNS lookups for remote messages
# See rsyslogd(8) for more details
RSYSLOGD_OPTIONS=""
So, this was basically a no-op -- rsyslog was being run with no options
by default. Because I hadn't needed to customise it, that meant that I
could rewrite the upstart script so that the whole "script...end script"
block was replaced by
exec rsyslogd
If the problem was the script block not playing well with the "limit
nofile" option, then you'd think that would work. But it didn't.
By this point (5 days in, working on this intermittently) I was
clutching at straws, and as a last ditch attempt I tried reverting back
to the original upstart script (so, no "limit nofile" bit, and the
original "script...end script" stuff sourcing the file from
/etc/default), and put a call to "ulimit -n 16384" inside
/etc/default/rsyslog, and restarted rsyslog.
It worked.
This seems really weird to me. I can sort-of-kind-of see why the normal
"limit nofile" upstart thing might not work, given that rsyslog forks
and daemonises itself, though it seems odd that it does work for other
processes (and there's a "expect fork" line in there to tell it to
handle that kind of behaviour).
But the fact that the ulimit has to be in /etc/default/rsyslog
completely baffles me. If it works when it's in a file that is sourced
into the script block, then why does it not work inside the script block
directly? I'm pretty sure that in normal shell commands, using "." is
basically equivalent to running the commands in the file inside your
current shell, one-by-one...
Anyway, I hope the solution is useful to someone out there, and that the
(somewhat rambling) explanation is of interest to someone too. And if
anyone does know the reason why it works in /etc/default/rsyslog but not
in the script block, or knows a better solution to the problem, I'd love
to hear!
All the best,
Giles
--
Giles Thomas <[email protected]>
PythonAnywhere: Develop and host Python from your browser
<https://www.pythonanywhere.com/>
A product from PythonAnywhere LLP
17a Clerkenwell Road, London EC1M 5RD, UK
VAT No.: GB 893 5643 79
Registered in England and Wales as company number OC378414.
Registered address: 28 Ely Place, 3rd Floor, London EC1N 6TD, UK
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
