If you're messing with the Mordred stuff, do you want to consider a simple statement cache so that the PreparedStatements aren't recompiled all the time? It may not make a big difference considering the amount of IO involved - but you never know.
Steve > -----Original Message----- > From: Noel J. Bergman [mailto:[EMAIL PROTECTED]] > Sent: Tuesday, February 11, 2003 8:35 AM > To: James Users List > Subject: RE: RE : Really slow spooling again... :( > > > Diego, > > At this time, there are no plans to replace mordred in James > v2.1. Those are some of the things that will be part of James v3. > > I spent quite a bit of time with mordred yesterday. One > thing that I found was that it really helps to add > autoreconnect=true to the MySQL URL. I had a spool with over > 200,000 items in it. Running James with no SMTP connections, > so only the spool is using JDBC, I could not run for long > before getting exceptions. With the auto-reconnect support, > I was able to process the entire spool at a rate of roughly > 1667 messages per minute without changing mordred at all. > > I would like to see spool processing much faster than that, > but where I am at in my testing is this: if I don't rate > limit postal many of the messages accummulate in the queue. > If I rate limit postal to 600 messages per minute, I can run > unimpeded for hours with 5% free CPU. Since normally I have > 0% free CPU during a load test, I am figuring that for this > system, 600 MPM is about the limit. That is with the spooler > performing work. With the spooler discarding messages at the > top of root, I can go about 1600 MPM, and wit the spooler > disabled, I seem to recall about 3000 MPM. So there appears > to be considerable room for improvement in the spooler. > > I was planning to make some tweaks to mordred, and perhaps > there are some changes we could make to the table structure, > but right now the biggest improvement in reliability for me > over the stock config.xml was to add the autoreconnect=true > parameter to the MySQL URL. > > Ultimately, I suspect that we'll have to revisit the spooling > mechanism, and possibly do some rate limiting on the incoming > connection based upon the spooler. > > --- Noel > > -----Original Message----- > From: Diego Castillo [mailto:[EMAIL PROTECTED]] > Sent: Tuesday, February 11, 2003 4:30 > To: 'James Users List' > Cc: Diego Castillo, INEXBEE > Subject: RE : Really slow spooling again... :( > > > Hi Noel, > > I have seen your mail on Mordred and some issues with JDBC > spool, which you seem to have corrected in latest build. > > I am experiencing myself some problems on heavy load > conditions (more than 5 mails per second). MySQL gets tired > and starts responding slowly. This slows James down, so the > spool queue grows and MySQL gets even slower. At some point > it takes really long to take a mail out of the spool (>60s), > and Mordred fires a DB connection timeout. Then it gets what > looks to me like a deadlock and James hangs (see stack trace > below). I need to restart James and OPTIMIZE TABLE spool, > following Danny's advice. > > Is this the issue you have corrected in 2.1.1a7? I am using > 2.1. I am looking forward searchable datasources via JNDI, > which Serge seems to have added to HEAD. Will this be part of > the 2.1.1 final release? > > Regards, > > > Diego > > > *** Stack trace *** > > database-connections.maildb: java.lang.Throwable > at org.apache.james.util.mordred.PoolConnEntry > .lock(PoolConnEntry.java:84) > at org.apache.james.util.mordred.JdbcDataSource > .getConnection(JdbcDataSource.java:127) > at org.apache.james.mailrepository.JDBCSpoolRepository > .loadPendingMessages(JDBCSpoolRepository.java:228) > at org.apache.james.mailrepository.JDBCSpoolRepository > .getNextPendingMessage(JDBCSpoolRepository.java:203) > at org.apache.james.mailrepository.JDBCSpoolRepository > .accept(JDBCSpoolRepository.java:137) > at org.apache.james.transport.mailets.RemoteDelivery > .run(RemoteDelivery.java:577) > at java.lang.Thread.run(Thread.java:536) > > -----Message d'origine----- > De : Noel J. Bergman [mailto:[EMAIL PROTECTED]] > Envoye : mardi 11 fevrier 2003 01:21 > A : James Users List > Objet : RE: Really slow spooling again... :( > > Kenny, > > Try adding some additional threads to RemoteDelivery, but > unless you also configured RemoteDelivery's output spool for > JDBC, do *not* do it unless you are using v2.1.1a7 or later, > due to a synchronization error that I found and fixed. > > Yes, the record at openrbl.org doesn't look good for you. > Are you a colo, or do you need a host? > > --- Noel > > -----Original Message----- > From: Kenny Smith [mailto:[EMAIL PROTECTED]] > Sent: Monday, February 10, 2003 18:31 > To: James Users List > Subject: Re: Really slow spooling again... :( > > > Hi Noel et al, > > It appears that connections from my server are getting denied > by hotmail and yahoo (my ISP just got labeled as a spammer... > I'm trying to find a new ISP), so I'm getting a lot of > timeouts and a lot of retries. I can only assume that it's > just going through the spool really slowly because it takes a > long time to timeout. :/ > > Me need new ISP. *sigh* > > Kenny > > Noel J. Bergman wrote: > > > Kenny, > > > > I have run tests most nights lately, each about 1 million messages. > They > > run through the spooler until they encounter a matcher that discards > them > > all. The tests average about 1400 messages per minute on > the 400 mhz > > Celeron test bed. > > > > The test basically exercises the heck out of the SMTP > handler and the > > spooler. The spooler is configured in mysql as: > > > > > > > > > > > > The data-source is the standard entry for MySQL, except that the > > database is "test" on that machine. > > > > What database (type and version) are you using? One thing that I > found a > > while ago was that on very rare occassions, I could go into > MySQL, do > a > > manual SQL query, and notice an excessive amount of time > (let's say 1 > > second instead of 0.006). Restarting MySQL (and James because of > > mordred) cleared > > that up. > > > > The fact that they are in transport is insufficient. What > you need to > > do is see where in the processor they are. Look for entries of the > > kind: > > > > Checking with > > Servicing by > > > > Those should tell us specifically where in the transport the message > has > > gotten, and the timestamps will tell us how long it is > taking to move > > through the spool. > > > > --- Noel > > > > -----Original Message----- > > From: Kenny Smith [mailto:[EMAIL PROTECTED]] > > Sent: Monday, February 10, 2003 13:10 > > To: [EMAIL PROTECTED] > > Subject: Really slow spooling again... :( > > > > Hi all, > > > > I'm having all of my mail backup in the spool tables (using > JDBC) with > a > > message_state of 'transport'. I've turned up my logging to > DEBUG, but > > I'm not seeing anything in the logs that looks relevant to slow > > performance. I sent 230 email 2 hours ago and there are still 190 in > the > > spool/transport waiting to get sent. > > > > I saw the recent conversation about indexes on the spool > table causing > > slow downs after a while, so I stopped James, dumped the table, > removed > > the indicies, put the table back in and started James up. Nothing > > appears to have changed. > > > > I'm running James 2.1 with jdk1.4 on Solaris. > > > > Any help is appreciated. > > > > Kenny Smith > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
