I have a few more things to add now with a little more testing.
I let my gateways backup on E-mail so that I could slam Declude, and
here's what happens.
When Declude is not hitting it's THREADS setting, it waits until the
work folder is empty before moving in a new batch. When Declude is
hitting the THREADS setting, it seems to move in messages just as
quickly as the threads are freed up. As soon as the proc folder no
longer contains more messages than your THREADS setting, Declude again
starts waiting until the work directory is empty before moving in
another batch. Here's a graph of one of these periods:
So while hitting your thread limit, it seems to be able to reach
virtually full server capacity, but it performs much differently when
you are under your THREADS capacity. This suggests that it is a bad
idea to have your THREADS setting too high, though this is only because
of the strange way that Declude acts.
It should be able to do almost 100% under these circumstances, but it's
not optimal behavior.
Matt
Matt wrote:
I figured that i would just start a new thread for this instead of
adding to the old one.
This was the first time that I have used a version of Declude that runs
as a service. I'm unfortunately surprised and disappointed at how it
handles things, but a lot more makes sense now.
In a nutshell, what happens is now Declude batch processes messages,
and waits for every message from a batch to complete processing before
it grabs a new batch. The time that it takes for a batch to process is
variable, and mostly depends on virus scanning and DNS
timeouts/slowness. Some batches complete in 10 seconds, but some take
more than a minute. The one that I noticed that took over a minute was
the result of 3/4 of that time spent on the last message which needed
to be virus scanned, and the message was large so it took a while to be
virus scanned.
The problem with this architecture is that when it moves a batch of
messages into the work folder for processing, it quickly pegs the
processor at 100% as it launches all of the threads, but most messages
go through all of the steps quickly so the processors sit almost idle
while it is waiting for DNS lookups and virus scanning (which I do
after JM). Here's a sample of my combined CPU graph showing the
pattern that this creates:
This is actually the most optimal pattern, but in one where there is a
significant delay in one message can go on for over a minute at lower
than 5% CPU utilization while it waits for one message that requires
extra work. During peak times on my server, I am seeing between 10 and
40 messages being moved to "work" at one time.
The net result of this is that my server will only handle less than
half the number of the number of messages that it could if I could
actually ride 100% and launch threads as messages came in instead of in
threads. Under this architecture, there is nothing that I can do about
this. I also don't like the idea of hitting 100% so frequently as it
does since running at 100% causes a much increased chance of
instability.
This architecture also explains why so many posted here about E-mail
backups. Their default settings would move in less messages than they
were collecting in the proc folder while waiting for things to be
processed. I'm sure that this graph doesn't look nearly as bad with a
vanilla install of Declude, but with 4 external tests, one of which
does multiple serial DNS lookups, and then the latency of having two
virus scanners run in serial means that finishing up a batch takes more
time, which is less time that the CPU is actually doing much.
Others have reported that Declude 3+ is faster than 2-. I run SNMP
monitoring of my server's CPU that takes one minute averages, and when
you average those out over an hour, my peaks are maybe 10% lower
relative to before (i.e. 32% instead of 35%). So it does apparently
work more efficiently to some extent, but there is no way that my
server could ever surpass 50% hourly averages when using the batch
processing, so my server's capacity is actually less than half of what
it was with 2.0.6.16. I am absolutely astonished that I haven't seen
anyone comment about this before.
What Declude needs to do is not wait for an entire batch to complete.
They should set a maximum thread limit, and then allow us to configure
the number of free threads to wait for before picking up new messages
to process. So instead of waiting 30 seconds for a virus scan to
complete on one message with 40 waiting in the proc folder, it should
start picking up messages as soon as it sees free threads.
I also immediately came across a bug with whitelisting. For some
reason WHITELIST IP is now being applied before IPBYPASS, and to
complicate matters, it won't log for JunkMail at LOGLEVEL MID when this
happens. In 2.0.6.16 and before, IPBYPASS was applied before any
processing based on the IP was done. I'm guessing that when they fixed
"PREWHITELIST ON" (which is broken in 2.0.6.16) they changed the
behavior and now whitelist before IPBYPASS for things like WHITELIST
IP, and this I am guessing also breaks the logging, though DEBUG
logging did show me the problem. I reported this, but only got a NAK
back so far.
I can work around the bug with whitelisting for now, but the batch
processing stuff is a show stopper for me in it's current format.
Matt
|
