On 18 Dec 2000, at 11:22, Joseph Scott wrote:
>
> On Sun, 17 Dec 2000, Dan Langille wrote:
>
> # Which list would be more appropriate for asking advice on designing a
> # mail processing strategy for FreshPorts 2 (i.e. processing all of
> # cvs-all, not just the ports)?
> #
> # I'm looking for recommendations and guidance on how to capture the
> # incoming messages and process them one at a time. As opposed to
> # starting a separate perl script for each message (which is the the
> # existing strategy and is usually fine, except when large numbers of
> # messages turn up in a short period of time).
>
> If you don't want to process a message the instant it comes in
> (via feeding it to a perl script or what ever) you'll need to setup some
> sort of queue, then have a cron job come through and process the queue.
Thanks. Since posting my original message, the proposed design has
drifted to this strategy (which I posted to the FreshPorts develop list last
night):
On Mon, 18 Dec 2000, Dan Langille wrote:
> ok folks, brain pick time: I'm thinking about redoing the FreshPorts
> mail-database interface. At present, each incoming email from cvs-all
> initiates a separate perl script. Not nice if you get say 100 emails all
> in 1 minute for some reason. Now I'm thinking of using procmail to
> dump each email to a separate file on disk. Then processing each
> file thus freeing up the MTA ASAP. Sound good so far?
>
> Suggestions? Ideas?
As for processing the files once they are on disk, I like the idea of
processing the messages ASAP. Therefore, I'd like to have a daemon /
script sitting there running all the time. This process is notified when a
new message arrives (in the above case, that would be part of the
procmail delivery process). This process would then "wake up" and
process all available message files one at a time. Perhaps moving each
processed file to an archive. When there are no more message files, it
would stop and wait to be notified again.
> If the problem is load then another approach would be to heavily
> nice(1) the perl script the is launched when a commit mail comes in.
That's already done. It's just the volume which can occur. If a large
number of messages arrive at once. Starting up 50 perl jobs on the box
can stress it a bit. It also makes better sense to process the
messages in the order in which they arrive rather than concurrently. A
side effect will be less change of database lockouts or conflicts during
updates.
cheers
--
Dan Langille
The FreeBSD Diary - http://www.freebsddiary.org/
NZ ADSL - http://www.unixathome.org/adsl/
NZ Broadband - http://www.unixathome.org/broadband/
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message