Phil Pennock wrote:
On 2011-05-19 at 12:35 +0100, Dominic Benson wrote:
On 19/05/11 11:50, Phil Pennock wrote:
New queuing system refers to an approach to scale up the spool directory
to something more queue-like, with segregated admin-defined queues (eg,
"big_freemail_provider_x"). This is because while Exim is excellent at
inbound mail, it doesn't always scale as well as some would like for
outbound mail which can't be immediately delivered. Nothing has been
done on this. Patches welcome.
Is this AKA bug 336? It sounds quite interesting, so I think I might
have a look at making some inroads into the problem.
If there are are any notes/thoughts about behaviour/config/use it would
be handy to hear them.
I wasn't aware of 336, but yes it's related.
At present, there's split_spool_directory, which divides things up with
one level of hashing, and then some people script their own queue-runner
launchers, running in parallel over sub-trees of the split spool instead
of having the Exim daemon launch runners over everything, which compete
with each other.
Nothing more specific was discussed, that I either recall or find in the
minutes; we all understood the general problem.
Known bound, yes.
Problem?
Not sure.
I stand on the position that WHEN one is loading Exim so heavily that it
HITS that sort of bound, one has far too many eggs in one basket
downtime-risk-wise, and should split that load over multiple Exim (free)....
... on multiple separate boxen (generally cheap). Or at least usually
'cheapER' than answering complaints from such a huge user base - or
rushing to a remote data center.
If we use a new sub-directory of spool_directory to hold named queues,
then previous Exim installs won't know of the content, but it should be
fairly easy to script a rollback tool which recombines queues into the
original queue.
Thinking briefly (while tired and prone to mistakes):
You'd need a way to declare "move this message to this other queue",
.... or just a small critter managing ephemeral softlinks, perhaps?
perhaps a way to restrict Routers to only apply to messages in certain
queues (seems optional as a performance optimisation, *could* be useful
for some more sophisticated Router rule bifurcations) and either a way
to control counts of queue-runners per named queue or an API to do so.
Probably do-able already with a bit of metal bending.
For sure routers have no problem delivering wherever admin-config
directed, and retrieval is just the other side of the same coin should
it be seen useful.
Probably best to have a global config option defining a list of allowed
queue names, so that a typo doesn't end up putting 100k messages into
"gmial" where they sit idle. ;)
Worse has happened.
Left a delimiter off the front of a complex SQL call once. Messages
disappeared for a while.
Took 'ls' to the mailstore.
Found a new user with the path to Maildir not the *result* of the SQL
returned from the DB...
.. but rather the full-text of the unbounded SQL call *itself* ..as the
directory name.
:-(
Beyond that, I think there's a lot of freedom for the implementor to
make their own decisions. :) Have at it!
Aye. 'Enough rope ....'
Bill
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
