At 09:59 AM 10/5/2007, Tobias Kreidl wrote: >So, if you remove the "LIMIT" restriction, it then just runs longer, >with no other adverse effects?
correct >That aside, I assume you want to run the >cleanup application at night because the load is generally a lot lower. correct, and at night nobody really relies on the secondary database server (see below) > >Just to get an idea what you're up against, how many messages are we >talking about per day and what is the configuration of your server in >terms of CPU(s), memory and disks that results in its working so hard to >keep up? This might be more complicated of an answer than expected, but here it goes. Messages/Day = 3.5million The servers which the policyd daemons run on do not have any issues with load and aren't any part of the delay. The issues arise from one of the database servers. We have a multi-master replication setup. The primary server is 2ghz opteron, 3G ram, mutiple raid1 arrays to distribute IO load. This db server takes care of the cleanup in seconds. The secondary db server receives the replicate cleanup command shortly thereafter and is not quite as beefy. 1ghz athlon, 640M ram, single raid array. The issue is completely IO load on the secondary server. The cleanup requests take roughly 30 minutes to complete, during which time all other replicated commands fall behind. More ram and perhaps more disks could solve the IO load for now; however, a much simpler and cheaper solution is to change the cleanup script to run once per day rather than hourly during an hour when nobody cares if the replication server is behind. Even if it takes an hour to run once per day instead of 30 minutes every hour, that is much better IMO. Thanks, - Nate >Nate wrote: > > I've noticed that under a high volume mail server using policyd, with > > the cleanup app running each hour, the operation can take some time > > to complete, which bogs down the database server considerably. I > > would like to change the cleanup script to run once daily instead in > > the middle of the night. > > > > Although, in the source code all the cleanup queries have LIMIT > > 100000 on them. Not a problem when running it hourly, but daily this > > could cause issues becuase the database may grow faster than 100000 > > entries per day. > > > > example line 74 of cleanup.c: "DELETE QUICK FROM triplet WHERE > > _datelast < %d LIMIT 100000", > > > > The limit command seems counterproductive. I can remove it from the > > source of my installation easy enough, but I would suggest it be > > removed from the distributable source as well unless there's a great > > reason to keep the limits. > > > > - Nate > > > > > > ------------------------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. > > Still grepping through log files to find problems? Stop. > > Now Search log events and configuration files using AJAX and a browser. > > Download your FREE copy of Splunk now >> http://get.splunk.com/ > > _______________________________________________ > > policyd-users mailing list > > policyd-users@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/policyd-users > > > > > >------------------------------------------------------------------------- >This SF.net email is sponsored by: Splunk Inc. >Still grepping through log files to find problems? Stop. >Now Search log events and configuration files using AJAX and a browser. >Download your FREE copy of Splunk now >> http://get.splunk.com/ >_______________________________________________ >policyd-users mailing list >policyd-users@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/policyd-users ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ policyd-users mailing list policyd-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/policyd-users
