Diego,
At this time, there are no plans to replace mordred in James v2.1. Those
are some of the things that will be part of James v3.
I spent quite a bit of time with mordred yesterday. One thing that I found
was that it really helps to add autoreconnect=true to the MySQL URL. I had
a spool with over 200,000 items in it. Running James with no SMTP
connections, so only the spool is using JDBC, I could not run for long
before getting exceptions. With the auto-reconnect support, I was able to
process the entire spool at a rate of roughly 1667 messages per minute
without changing mordred at all.
I would like to see spool processing much faster than that, but where I am
at in my testing is this: if I don't rate limit postal many of the messages
accummulate in the queue. If I rate limit postal to 600 messages per
minute, I can run unimpeded for hours with 5% free CPU. Since normally I
have 0% free CPU during a load test, I am figuring that for this system, 600
MPM is about the limit. That is with the spooler performing work. With the
spooler discarding messages at the top of root, I can go about 1600 MPM, and
wit the spooler disabled, I seem to recall about 3000 MPM. So there appears
to be considerable room for improvement in the spooler.
I was planning to make some tweaks to mordred, and perhaps there are some
changes we could make to the table structure, but right now the biggest
improvement in reliability for me over the stock config.xml was to add the
autoreconnect=true parameter to the MySQL URL.
Ultimately, I suspect that we'll have to revisit the spooling mechanism, and
possibly do some rate limiting on the incoming connection based upon the
spooler.
--- Noel
-----Original Message-----
From: Diego Castillo [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, February 11, 2003 4:30
To: 'James Users List'
Cc: Diego Castillo, INEXBEE
Subject: RE : Really slow spooling again... :(
Hi Noel,
I have seen your mail on Mordred and some issues with JDBC spool, which
you seem to have corrected in latest build.
I am experiencing myself some problems on heavy load conditions (more
than 5 mails per second). MySQL gets tired and starts responding slowly.
This slows James down, so the spool queue grows and MySQL gets even
slower. At some point it takes really long to take a mail out of the
spool (>60s), and Mordred fires a DB connection timeout. Then it gets
what looks to me like a deadlock and James hangs (see stack trace
below). I need to restart James and OPTIMIZE TABLE spool, following
Danny's advice.
Is this the issue you have corrected in 2.1.1a7? I am using 2.1. I am
looking forward searchable datasources via JNDI, which Serge seems to
have added to HEAD. Will this be part of the 2.1.1 final release?
Regards,
Diego
*** Stack trace ***
database-connections.maildb: java.lang.Throwable
at org.apache.james.util.mordred.PoolConnEntry
.lock(PoolConnEntry.java:84)
at org.apache.james.util.mordred.JdbcDataSource
.getConnection(JdbcDataSource.java:127)
at org.apache.james.mailrepository.JDBCSpoolRepository
.loadPendingMessages(JDBCSpoolRepository.java:228)
at org.apache.james.mailrepository.JDBCSpoolRepository
.getNextPendingMessage(JDBCSpoolRepository.java:203)
at org.apache.james.mailrepository.JDBCSpoolRepository
.accept(JDBCSpoolRepository.java:137)
at org.apache.james.transport.mailets.RemoteDelivery
.run(RemoteDelivery.java:577)
at java.lang.Thread.run(Thread.java:536)
-----Message d'origine-----
De : Noel J. Bergman [mailto:[EMAIL PROTECTED]]
Envoye : mardi 11 fevrier 2003 01:21
A : James Users List
Objet : RE: Really slow spooling again... :(
Kenny,
Try adding some additional threads to RemoteDelivery, but unless you
also
configured RemoteDelivery's output spool for JDBC, do *not* do it unless
you
are using v2.1.1a7 or later, due to a synchronization error that I found
and
fixed.
Yes, the record at openrbl.org doesn't look good for you. Are you a
colo,
or do you need a host?
--- Noel
-----Original Message-----
From: Kenny Smith [mailto:[EMAIL PROTECTED]]
Sent: Monday, February 10, 2003 18:31
To: James Users List
Subject: Re: Really slow spooling again... :(
Hi Noel et al,
It appears that connections from my server are getting denied by hotmail
and yahoo (my ISP just got labeled as a spammer... I'm trying to find a
new ISP), so I'm getting a lot of timeouts and a lot of retries. I can
only assume that it's just going through the spool really slowly because
it takes a long time to timeout. :/
Me need new ISP. *sigh*
Kenny
Noel J. Bergman wrote:
> Kenny,
>
> I have run tests most nights lately, each about 1 million messages.
They
> run through the spooler until they encounter a matcher that discards
them
> all. The tests average about 1400 messages per minute on the 400 mhz
> Celeron test bed.
>
> The test basically exercises the heck out of the SMTP handler and the
> spooler. The spooler is configured in mysql as:
>
>
>
>
>
> The data-source is the standard entry for MySQL, except that the
> database is
> "test" on that machine.
>
> What database (type and version) are you using? One thing that I
found a
> while ago was that on very rare occassions, I could go into MySQL, do
a
> manual SQL query, and notice an excessive amount of time (let's say 1
> second
> instead of 0.006). Restarting MySQL (and James because of mordred)
> cleared
> that up.
>
> The fact that they are in transport is insufficient. What you need to
> do is
> see where in the processor they are. Look for entries of the kind:
>
> Checking with
> Servicing by
>
> Those should tell us specifically where in the transport the message
has
> gotten, and the timestamps will tell us how long it is taking to move
> through the spool.
>
> --- Noel
>
> -----Original Message-----
> From: Kenny Smith [mailto:[EMAIL PROTECTED]]
> Sent: Monday, February 10, 2003 13:10
> To: [EMAIL PROTECTED]
> Subject: Really slow spooling again... :(
>
> Hi all,
>
> I'm having all of my mail backup in the spool tables (using JDBC) with
a
> message_state of 'transport'. I've turned up my logging to DEBUG, but
> I'm not seeing anything in the logs that looks relevant to slow
> performance. I sent 230 email 2 hours ago and there are still 190 in
the
> spool/transport waiting to get sent.
>
> I saw the recent conversation about indexes on the spool table causing
> slow downs after a while, so I stopped James, dumped the table,
removed
> the indicies, put the table back in and started James up. Nothing
> appears to have changed.
>
> I'm running James 2.1 with jdk1.4 on Solaris.
>
> Any help is appreciated.
>
> Kenny Smith
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
