Re: conf-split
Peter van Dijk [EMAIL PROTECTED] writes: On Thu, Jan 25, 2001 at 02:12:32AM +, James R Grinter wrote: [snip] Indeed, qmail already uses a split queue/mess/ directory structure and it was a bit of an omission to assume that there would never be a surge of mail in one go (VERP list expansion is definitely good for creating this situation) and thus many messages in todo/ at once. VERP expansion happens on delivery, not on queue injection, unless you are doing something very wrong. It's always good to question and investigate what is happening - thanks to Peter for the prompting - the answer seems to be that the "majordomo-inject" script we've been using since 1998 was indeed expanding upon queue injection (it was doing the VERP itself.) Anyone out there using this - *do* switch to mjinject instead - Giles Lean and Russ Allbery's replacement script. James. (Only 2 and a half years to spot and nail the problem. Not bad...)
Re: conf-split
On Wed, Jan 24, 2001 at 12:44:06PM +0100, Peter van Dijk wrote: Are there any negative sideeffects of setting conf-split too high? Yes there are (answering my own post). Scanning todo/ takes longer, if you are using the big-todo patch, because every subdir has to be scanned, instead of just one dir. Where *is* the benefit in the big-todo patch? Greetz, Peter.
Re: conf-split
On Wed, Jan 24, 2001 at 06:59:26PM +0100, Peter van Dijk wrote: Yes there are (answering my own post). Scanning todo/ takes longer, if you are using the big-todo patch, because every subdir has to be scanned, instead of just one dir. Where *is* the benefit in the big-todo patch? I think the benefit is with OSs having poor directory access routines. As directories are scanned on a linear basis access is faster scanning two small directories than one very big one. Also modifications in the small directories (adding/removing files) will be faster. And (not sure about that, though) the first level directory is held in the filesystem cache as it has lots of accesses but does (usually) not change. However IIRC this patch is mostly (only?) a benefit with Linux' ext2 filesystem. \Maex
Re: conf-split
Markus Stumpf [EMAIL PROTECTED] writes: I think the benefit is with OSs having poor directory access routines. Such as Solaris/UFS (and, to my knowledge, all the *BSDs that use UFS have not improved this much either.) As directories are scanned on a linear basis access is faster scanning two small directories than one very big one. Also modifications in the small directories (adding/removing files) will be faster. UFS code, when looking for a filename, checks for a matching size, then a matching first letter, second letter, and the rest: so it's sub-optimal to have all your files in a single directory with the same filename length and starting letters (as everyone running INN eventually found out.) Each stat() or open() is a linear scan through the directory to find the file. And (not sure about that, though) the first level directory is held in the filesystem cache as it has lots of accesses but does (usually) not change. Whilst qmail-queue is scanning over todo/ I'd expect it to remain in the various caches, with or without splitting up the directories. Indeed, qmail already uses a split queue/mess/ directory structure and it was a bit of an omission to assume that there would never be a surge of mail in one go (VERP list expansion is definitely good for creating this situation) and thus many messages in todo/ at once. Further more, when there are many messages in todo/ the processing overwhelmes the rest of the work that qmail-queue normally does and slows down outbound deliveries to a crawl. All it takes is one of the first people on a list to reply quickly and then you get another list delivery, and yet more messages in todo/. Very messy business. I've just started working on trying to collect (and thereby graph) as much data as I can on qmail's operation - but one thing that isn't recorded in the logs is the time between todo/ creation by qmail-inject and 'new msg'. If anyone has previously made changes to the code to accomplish this, could they let me know and/or give me some pointers? Otherwise I'll have to go digging. (Any patches to qmail-smtpd to log useful things would also be of help to me: I've seen, and am looking at Maex's code.) James.
Re: conf-split
On Thu, Jan 25, 2001 at 02:12:32AM +, James R Grinter wrote: [snip] Indeed, qmail already uses a split queue/mess/ directory structure and it was a bit of an omission to assume that there would never be a surge of mail in one go (VERP list expansion is definitely good for creating this situation) and thus many messages in todo/ at once. VERP expansion happens on delivery, not on queue injection, unless you are doing something very wrong. Greetz, Peter.
