Re: [vchkpw] smtp after pop
1. how come i'm able to use the same smtp on two different machines on the home network without having to authenticate for smtp on the imap account? It is possible that you're using a NAT router/gateway between your home network and your ISP? If so, the NAT address on the WAN side of your router serves both of your home computers. That address is the one that is known to the Qmail/Vpopmail server, not the different internal addresses. To the mail server, you're coming from the same address. To see ensure that your mail relay filters are working, go here: http://www.abuse.net/relay.html and follow the instructions for the anonymous test. -- Eric Ziegast
Re: [vchkpw] vpopmail and nfs
I assume CentOS is some kind of Linux? The email below is geared toward
Linux.
File busy try again later!
Failed while attempting to update_file() the assign file
Error. Failed to add domain to assign file
Error: Could not update file
Let's see if we can use the source to figure out what's wrong.
In vpopmail.c, the code is trying to append a line to the assign file.
To do this with FILE_LOCKING defined, the update_file routine needs
to open and lock assign.lock. If it doesn't, it complains:
Failed while attempting to update_file() the assign file
The update_file routine calls get_write_lock to do its dirty work.
Here's the routine in file_lock.c:
int get_write_lock( FILE *fs )
{
int try = 0;
while(write_lock(fileno(fs), 0, SEEK_SET, 0) 0)
{
if (errno == EAGAIN || errno == EACCES || errno == ENOLCK )
{
/* there might be other errors cases in which
* you might try again.
*/
if (++try MAX_TRY_RLOCK) {
(void) sleep(2);
continue;
}
(void) fprintf(stderr,File busy try again later!\n);
return(-1);
}
return(-2);
}
return(0);
}
The write_lock() routine is a wrapper for fcntl(). If you look at
the fcntl(2) man page, it lists many reasons why it would fail.
Based on your second email, it's not permissions. Some other program
must have a lock on the file and is not letting go, or there's some
other odd problem (NFS? extended attributes?).
I have a few suggestions to help debugging:
1. Run strace -f /home/vpopmail/bin/vadddomain ...etc
Look at the output right before the printf of the error messages
for the return value from fcntl.
2. Run lsof assign.lock after it fails to see if there are any
other processes that have the file open. This would be done on
both the client and the server.
3. Add the following line before the line that has ++try MAX_TRY_RLOCK:
(void) perror(Debug write_lock: )
Then recompile vpopmail, install it, and try it again (don't forget
to make backups!). You'll hopefully get a reason for the locking
failure printed when it happens again.
4. Restart the NFS client and the NFS server and try again (stale lock).
Also, something you might need to do is make sure your NFS server
doesn't clobber UID names. I noticed in one of your emails that
assign.lock was owned by nfsnobody. The export options on the
NFS server should use no_all_squash,no_root_squash,async,rw.
On most distros, no_all_squash is the default option, but your
distro might have it set the other way.
Other NFS stuff:
1. The client needs to use NFS V3 for fcntl locking to work.
See this article:
http://www.quepublishing.com/articles/article.asp?p=23618seqNum=4
2. Is nfslockd running on the client?
3. Is the NFS client and NFS server running the same OS? If not,
could there be some compatability issues?
Hope this helps.
--
Eric Ziegast
PS: If all else fails, install vpopmail on the NFS server and run the
administrative commands (vadddomain, vadduser, etc.) on the NFS server.
Patient: Doctor, when I hit my hand to make it spin around, it hurts.
Doctor: Stop doing that, and you'll be fine. That'll be $200.
Re: [vchkpw] vpopmail problem
Can anyone suggest me solution why user directoy are created in /home/vpopmail/domains/domainname/0/ - folder. This is explained well in the README file that comes with vpopmail. Look for user directory structure. Imagine test.com when it gets up to 500 or 5000 or 5 users. To have all of the users in one directory is not efficient on all filesystems so vpopmail automatically uses hashing to break user directories out into subdirectories after the first 100 users. If I had a script to create vadduser user1 up to user25000, I might find the user hashing to look like... /home/vpopmail/domains/DOMAIN/__/Maildir user1 user2 ... user99 user100 0/user101 0/user102 ... 0/user199 0/user200 1/user201 1/user202 ... 9/user999 9/user1000 A/user1001 ... Z/user3600 a/user3601 ... z/user6200 0/0/user6201 ... 0/z/user12400 1/0/user12401 ... 1/z/user24800 2/0/user24801 ... 2/1/user25000 Instead of assuming that a user's Maildir is in the directory cd /home/vpopmail/domains/$DOMAIN/$USER/Maildir, Vpopmail administrators typically use cd `vuserinfo -d [EMAIL PROTECTED], or even safer, cd `vuserdir [EMAIL PROTECTED]. #!/bin/sh # WARNING: untested code - use at your own risk # # vuserdir - print out the vpopmail directory for a user #or . if it does not exist. # # Syntax: vuserdir address # address=$1 if [ $address = '' ] then echo vuserdir: No address specified 12 echo . exit 1 fi dir=`vuserinfo -d $address` if [ $? = 0 -a -d $dir ] then echo $dir else echo vuserdir: Directory for $address not found: $dir 12 echo . exit 1 fi The hashing is controled by the OPEN_BIG_DIR define in vpopmail's config.h. If you never intend to use large user directories (*), then commenting this out the define before installing might avoid this behavior, or you can compile MAX_USERS_PER_LEVEL (in vauth.h) to be a really high number. Best practice, though, would have you use directory hashing just like everyone else. Imagine that someday someone else withh have to take over your qmail/vpopmail installation and worry about your local customizations. It becomes difficult for them to upgrade to the next version of vpopmail if they have to patch the code each time. The directory hashing is also used on domains. So if you add more than 100 domains, you'll see /home/vpopmail/domains/0. -- Eric Ziegast [EMAIL PROTECTED] (*) Back in the 80's someone said, No one will ever need more than 640 kilobytes of memory.
Re: [vchkpw] chkuser 2.0
Rick Macdougall asked
I didn't see anything in the docs or change logs specifically regarding
the mysql connection problem that sometimes crops up with vpopmail. Has
this been addressed ?
Antonio Nati replied:
Yes, chkuser 2.0 includes a new call, vauth_open(), that Rick Widmer told
is in vpopmail CVS, and will be able to return the status of connection.
As default this call is disabled, and should be enabled (uncommenting
#define CHKUSER_ENABLE_VAUTH_OPEN in chkuser_settings.h) when this call is
released.
Jeremy Kitchen added:
as a suggestion, would it be possible to have a definition placed into
vpopmail.h such as:
#define HAVE_VAUTH_OPEN
that way programs that link against vpopmail can support both methods
without any user intervention, and also this way, if say perhaps
the postgres code doesn't have vauth_open, and the mysql does, etc,
so someone linking against it need not worry about it :)
While the current chkuser.c might compile fine against a vpopmail
install with Sybase or Postgres, it doesn't work for MySQL because
vauth_open is defined only in vpgsql.c and vsybase.c.
To get around this, I replaced the following line in chkuser.c:
if (vauth_open () == 0) {
with:
#ifdef USE_MYSQL
#ifdef MYSQL_REPLICATION
if (vauth_open_read () == 0) {
#else
if (vauth_open_update () == 0) {
#endif
#else
if (vauth_open () == 0) {
#endif
It seems to work fine for me.
It might make more sense for vpopmail to just create a generic
vauth_open_read function for each supported auth method so that
external modules like chkuser.c would be able to use the right
semantics without getting specific about the details. Even in
the case of CDB (vcdb.c) the function would be defined as opening
the filehandle for reading the CDB file. ... just a thought.
--
Eric Ziegast
Re: [vchkpw] simscan test release: simplified scanner for clamav/spamassassin
Also I was wondering if there is a way to tell qmail-smtpd to exit with a 5xx error code rather than a 4xx error on spam or a virus? I am running qmail out of supervise, not that it should make any difference that I can see. Look at the end of qmail.c in the qmail distribution. The exit code of the child program (qmail-queue or $QMAILQUEUE) determines what error message is spit out in SMTP. The short answer is: Exit code 0 means successful delivery. Exit codes between 11 and 40 are permanent (5xx) errors. All other exit codes are temportary (4xx) errors. A longer answer is: Have your program exit with a specific exit code and then add a custom message in qmail.c to handle that code. For example: case 32: return Dmail server thinks this message is spam(#5.3.0); case 33: return Dvirus rejected(#5.3.0); case 34: return Dyou send us nothing but crap(#5.3.0); case 35: return Dyou are listed in the our blackhole list(#5.3.0); case 36: return Dleave us alone, jackass(#5.3.0); case 41: return Zif you are not a spammer, try again in 5 minutes(#4.3.0); -- Eric Ziegast
Re: [vchkpw] RedHatLinux 9 compatibility
[EMAIL PROTECTED] said: I want to install QMAIL + Vpopmail + courier-imap + Qmailadmin + LDAP + MySql + Spamassassin + clamav + Squirrelmail + stats (Isoqlog qmail MRTG) under Linux 9. Is every thing compatable with linux 9 where can i get good howto. I need step by step installation. also i want to host multiple mail sites on a single server. please help me in this regard Red Hat 9 is linux, and every software that you named is linux, it's redhatlinux 9 compaible. There's a lot of documentation about how to set up this software, you can find it on www.qmail.org . That's not quite true. I'll give you this: There _is_ documentation to take care of everything. It's only through experience that you learn/realize which documentation you need to look through to make sure everything works well without trial and error. The only compatability problem I'm aware of is Qmail. Here's something I googled that will help: http://forum.psoft.net/archive/index.php/t-7499.html There's a web site called lifewithqmail.org that goes through alot of the step-by-step process for Qmail if you're not familiar with how to install/administer DJB tools (eg: daemontools, tcpserver). I have a preference to use the MySQL 4 downloaded from www.mysql.com instead of the MySQL 3 supplied by RedHat. When doing things like backups or replication, MySQL 4 is better. There are also a couple books: Qmail - www.ora.com The Qmail Handbook - www.apress.com I like the newer O'reilly book better. If you are getting paid by your company to install everything, and if you make more than $6/hour, it will be faster, more cost-effective, and less problematic to have someone from Inter7 install Web Q for you within a few hours. They install all of this stuff for a living and get it right the first time. You can then go through installing everything yourself at your leisure as an educational exercise. They leave you their souce code behind so you can see the differences between stock code and their modifications. -- Eric Ziegast (no, I don't work for Inter7)
Re: [vchkpw] forwording mail's from qmail to ______ server
[EMAIL PROTECTED] wrote: We are shifting one of mail server from qmail to Exchange server for testing, ... my sympathies. You'll be back. so how can i forward mails from qmail mail server to Exchange server. Look at the qmail-remote manual page. http://qmail.geto.net/man/man8/qmail-remote.html If it's just POP mailboxes (not IMAP with subfolders), you can push the messages out using qmail-remote. Below are examples of how to remail messages in people's folders. Your script may not end up being exactly the same, but it should give you an idea as to qhat you need to do. #!/bin/sh PATH=/var/qmail/bin:/home/vpopmail/bin:$PATH # or whatever export PATH REMOTE=NEWSERVER.YOUR.DOMAIN [EMAIL PROTECTED] USERS=[EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] for user in $USERS; do dir=`vuserinfo -d $user` for subdir in new cur; do cd $dir/Maildir/$subdir || continue for msg in *; do qmail-remote $REMOTE $ADMIN $user $msg done done done If you just have one domain using qmail and not vpopmail, just change directories to the domain and run: #!/bin/sh PATH=/var/qmail/bin:$PATH # or whatever export PATH REMOTE=EXCHANGE.YOUR.DOMAIN [EMAIL PROTECTED] for user in * do for subdir in new cur; do cd $dir/Maildir/$subdir || continue for msg in *; do qmail-remote $REMOTE $ADMIN [EMAIL PROTECTED] $msg done done done If you need to preserve IMAP subfolders, it gets more complicated. You'd actually have to write (as Jeremy said) a Perl script using Net:IMAP to push the message into their new mailboxes. If it's just a few users, each user can add the new IMAP account and just drag/drop the messages into their new folders. -- Eric Ziegast
Re: [vchkpw] Filesystem question
I'm thinking that there are probably more efficient filesystems than ext3 for at least some of this, and have been thinking about using xfs (faster but still journaling), or maybe ext2 (very fast but no journaling) for at least the qmail queue. Does anyone have some recommendations for the filesystem to use on the partitions? I normally use reiserfs on the vpopmail home dir, and ext3 or ext2 on the qmail queue partition (due to old problems with reiserfs and qmail). I have recently installed a few systems with reiserfs on every partition and have seen no untoward problems. I did too. When using ReiserFS /w Qmail, don't forget to apply the patch. http://www.jedi.claranet.fr/qmail-link-sync.patch Details: http://www.jedi.claranet.fr/reisersmtp.html I've heard good things about both jfs and xfs with vpopmail home and qmail queue, but I haven't had time to play with them or benchmark them under heavy load. The one thing that I might like with XFS over otehrs is that it is the only filesystem with documented hooks for using LVM snapshots Has anyone ever used snapshots with Ext3 or ReiserFS? Snapshots are great becasue it's the fastest way to backup your mail partition against user errors (Help, I deleted my mailbox by mistake!) without adding stress/load to the mail server. -- Eric Ziegast
Re: [vchkpw] [semi-OT] which is more portable?
maybe I'll just make conf-vpopmail be the path to vpopmail's home directory and call it good. T'would be nice to be able to auto-detect it simply based on the user vpopmail was configured with (considering there are probably far less vpopmail implementations that use another user than vpopmail, than there There's no rule that says that the base vpopmail dir has to be the home dir of user vpopmail. You could compile a vconfig program first in C and use that program later in your install process. -- Eric Ziegast
Re: [vchkpw] [semi-OT] which is more portable?
Oops, sorry, didn't mean for that to go to the list. -- Eric Ziegast
Re: [vchkpw] NFS / Disk Access / Load Concerns on Vpopmail cluster
What we're seeing is that our network and RAID 5 IDE-based disk array on our central mail store server is not able to keep up with the 'client' servers doing the POP3, IMAP, Webmail, and SMTP legwork. I've found an interesting bottleneck with webmail. When people use POP or IMAP clients (Outlook, Mozilla, Opera, Thunderbird, etc.), the client application caches alot of the information locally and synchronizes occasionally with the server to see if there are new messages. Things like browsing and searching run eally fast because the user is utilizing the resources of their local PC to do most of teh work. With webmail, the session state is not saved nor cached, so with each new request, the mailbox can be rescanned. A relatively modest webmail application might only rescan all headers and show subject lines. A complex application might scan all content in a folder to present content more fully. Without anything to throttle back the webmail server, it's possible that the webmail server softwar can pound the mail spool server to death. I used to run a Qmail-based infrastructure for 4000 clients on a single slow machine without much memory. They used POP as their only pickup mechanism. We recently reimplemented on a Dell 1750 with two Xeon procs, alot of RAM and a GigE backend to a NetApp filer with 14 fast disks, and I STILL notice that the machine sometimes slowed down while people tried to read their 140MB mailboxes via webmail. sigh I put some bottlenecks on the search and retrieval algorithms of the webmail software to help protect the filer from a flood of queries, and we've been better since then. The power users with super-large mailboxes complain that it's slow, but now its a localized problem rather than a problem that affects everyone. Jeremy's comments are great for scaling the database, but it sounds to me that you're just maxed out on what you can serve over NFS. An SQL select might take at most a few kilobytes of data on the network whereas a webmail scan of a 30MB mailbox will take, well, 30MB. Doh! So what to do? Instead of the centralized NFS mail spool (where the central spool becomes the bottleneck), you might consider splitting the user base across several machines. Each machine would have its own RAID1 mail spool. Each machine would be responsible for its own Inbound SMTP and POP/IMAP/Webmail and use the local disk for the spool. Use lots of RAM for buffer cache to make sure your disk is hit less frequently. You might be able to centralize outbound SMTP. Once a machine fills up, you add another machine. This is one way to scale. The big boys in teh mailbox size wars (google, yahoo, hotmail) can't afford centralized storage for their mailboxes. Look for each to roll out racks of distribtuted storage where each storage server is a 1/2 U box with a couple large ATA disks in it. We might learn from this method of scaling. Before we take this costly step, what have you noticed for user / system loads before you start hitting the limits of your hardware? Yes. I serve 6000 users right now. They used to all be POP, and life was good. Now a significant percentage of my new customers use webmail, and I'm not happy with how my current web-based mail reading software scales. I may have to hack it alot to get it to perform well. Something that would help is if we rolled out spam/virus filtering out for everyone whih will cut 50% inbound mail and 10% viruses from being processed/stored/read and reread/reread/reread. BTW: I separate SMTP processing (/var/qmail local RAID1 fast SCSI with battery cache) from user mail spool storage (/home/vpopmail NFS mount to filer). Putting /var/qmail on the NFS server might be another source of overload. -- Eric Ziegast
[vchkpw] Possible enhancements to help protect qmail server reources against spam processing
invalid addresses. It's not to hard to log and summarize
the worst offenders, those that send 99% spam of 100 messages or more.
Once they do, you want to add them to your own blacklist. I hope to
create my own reactive system that utilizes spamassassin,
qmail-scanner-queue, tinydns and the soft rblsmtpd above to help my
server stay ahead of the flood instead of sink further into the depths
of mail lag.
--
Eric Ziegast
[EMAIL PROTECTED]
[EMAIL PROTECTED]
Patches so far
*** /home/inter7/vpopmail/vpopmail-5.4.0-spam2/vpopmail.c.orig 2004-06-10
02:34:54.0 -0700
--- /home/inter7/vpopmail/vpopmail-5.4.0-spam2/vpopmail.c 2004-05-20
02:37:54.0 -0700
***
*** 212,214
} else {
! fprintf(fs, | %s/bin/vdelivermail '' bounce-no-mailbox);
fclose(fs);
--- 212,214
} else {
! fprintf(fs, | %s/bin/vdelivermail '' [EMAIL PROTECTED], VPOPMAILDIR, domain);
fclose(fs);
*** /home/inter7/ucspi-tcp/ucspi-tcp-ssl-0.88/rblsmtpd.c.orig 2000-03-18
07:18:42.0 -0800
--- /home/inter7/ucspi-tcp/ucspi-tcp-ssl-0.88/rblsmtpd.c 2004-06-10
01:34:38.0 -0700
***
*** 177,180
! while ((opt = getopt(argc,argv,bBcCt:r:a:)) != opteof)
switch(opt) {
case 'b': flagrblbounce = 1; break;
--- 177,181
! while ((opt = getopt(argc,argv,bBcCt:r:a:s:)) != opteof)
switch(opt) {
+ case 's': flagrblbounce = 2; if (!pathexec_env(QMAILQUEUE,optarg)) nomem();
break;
case 'b': flagrblbounce = 1; break;
***
*** 193,197
if (flagwantdefaultrbl) rbl(rbl.maps.vix.com);
! if (decision = 2) rblsmtpd();
! pathexec_run(*argv,argv,envp);
strerr_die4sys(111,FATAL,unable to run ,*argv,: );
--- 194,198
if (flagwantdefaultrbl) rbl(rbl.maps.vix.com);
! if (decision = 2 flagrblbounce == 1) rblsmtpd();
! pathexec(argv);
strerr_die4sys(111,FATAL,unable to run ,*argv,:
Re: [vchkpw] Re: SMTP Auth HOWTO?
I know this is a shameless plug, but I'm a happy customer. Have Inter7 do a SugarBox install for less time/money than it takes to figure it out using online resources and googled howtos. I didn't have to second-guess or debug anything. Within 4 hours of the consultant logging in via SSH, I had SMTP-AUTH, POP-before-SMTP, SMTP/SSL, POP3, POP3/SSL, IMAP, IMAP/SSL, CRAM-MD5 and a complement of TinyDNS and SqWebMail all working together. Within another hour, he had MySQL replication and redundancy working. He left all the source code on my box so that I could make modifications and customizations later using make install and even build additional servers later. If you don't make a living installing Qmail/Vpopmail servers, it's less expensive and more practical to just let someone else do it. I've installed qmail/vpopmail from scratch before and believe that it can be a PITA to get done right. -- Eric Ziegast
[vchkpw] *.COM - 64.94.110.11
Verisign is breaking some peoples' spam filtering. Imagine that [EMAIL PROTECTED] is now a _valid_ email address. Many qmail/vpopmail users also use dnscache. Here's a patch for people who use dnscache to preserve the old (uninfected) behavior... http://tinydns.org/djbdns-1.05-ignoreip.patch -- Eric Ziegast
Re: [vchkpw] Big server
The client mail server would serve whatever combination I would like make a big server with qmail +vpopmail +mysql +procmail. I think in this structure: Server 1: Mx domain + smtp delivery +filters (Antispam, user filter(procmail) and antivirus) This server basically is the mail gateway of all domains, where is passed in the filters rules per domain and redirect all mails to server 2 Server 2: Pop3 accounts + mysql server Here is created all accounts. This schema is good for multiple domains? Based on my experience, I agree, but I might split Server2 into Server2 (delivery/storage/database) and Server3 (pop/imap/webmail servers for clients). I include more details below for one economical infrastructure I worked with. It's not a HOWTO, but knowing what someone else has done might help guide you instead of figuring it out from scratch. Another question is how do i do the message delivery the messages from server 1 to server2? Qmail! :^) In /var/qmail/control/smtproutes, set it so that all mail goes to Server2 (eg: :server2.mydomain.com). If you're fancy, you can try QMTP instead of SMTP. -- Eric Ziegast A sample large server environment (hundreds of domains, thousands of users) I once helped with: The MX record points to multiple cheap parallel inbound mail servers: - Single CPU PC at the best Price/Performance cost. I've found that one can build these for $300 each. You will find that when doing Virus/Spam scanning that the first bottleneck that you hit (out of CPU/memory/disk/network) is CPU. All of the regular expression searching on an e-mail message takes processing power. Assuming you have enough RAM, disk I/O would be the next bottleneck. I found a good balance AMD 1800+ motherboard /w 512MB PC133 RAM and 7200RPM IDE. Another option is investing in a very fast multi-processor Intel screamer with lots of RAM, but the cheap and disposable dervers are linearly scalable. - RAM depends on how many simultaneous connections you want to be allowed for Spam/Virus filtering. I used 512MB on a cheap system becasue RAM is cheap these days. I usually ran out of CPU before memory. If the OS uses any significant amount of virtual memory, you need more RAM. Run vmstat. If pi or po is above 0, you need more RAM or need to lower the number of simultaneous connections allowed by qmail (eg: concurrencyincoming in /var/qmail/control). The inbound server is your mail firewall and doesn't have time for paging to disk when the message load is high. - Hardware or softare RAID1 7200+ RPM IDE drives is sufficient. I have been told by a Linux integrator that Linux software RAID1 can be faster than the RAID1 provided by hardware controllers. If you have a budget for SCSI, use it. You need merely a 9GB drive in an inbound relay server anyway because the mail doesn't sit on the server. In fact, you may see a disk I/O improvement if you limit /var/qmail/queue to a 2GB partition of the hard drive. If you don't need the space, you don't need to have the disk head potentially cross the entire disk to find data. If you select a hardware RAID controller, prefer a controller that has non-volatile RAM or RAM /w a battery. This will allow the controller to use write-back mode on write and significantly reduce response time between the computer and the hard drives. - While I love OpenBSD and FreeBSD, I've used Linux for Qmail services because I've had other Linux-capable staff that could help administer the servers. Another advantage to Linux is ReiserFS. I have used ReiserFS on /var/qmail/queue partitions with success after applying the fsync patches. (http://www.jedi.claranet.fr/qmail-reiserfs-howto.html) ReiserFS performs well with thousands of files in a directory and allows you to keep the default hash value (23) for the spool directory. If using ufs (Solaris/BSD), consider compiling a queue hash value of some large prime number (like 101). If using Linux without ReiserFS, at least use ext3 instead of ext2 so that you can recover after a crash. If using Solaris, consider VxFS if you have the ability to use it. A standard fsck of a non-journaled filsystems used for qmail REALLY sucks. Aside: I don't export ReiserFS over NFS - just use it for the mail relays themselves. For vpopmail directories, I use filesystems that are known to be tried and tested in heavy read/write environments under NFS. I hope ReiserFS gets to this state, but at the time of my implementation, it was easier for me to use ext3 for vpopmail dirs. - I followed instructions for using QmailScanner /w SpamAssassin (spamc -f -c) and a Virus checker. I found QmailScanner to be quite inefficient and significantly rewrote it to not break up the message into a zillion pieces for its internal scanning. SpamAssassin (spamd) does that for you anyway
[vchkpw] Re: Spam Assassin implementation
News from the front lines: In a world that has mostly benign spam where spammers with real return addresses send messages to valid recipients, qmail-scanner has its place. You can easily tag spam qith qmail-scanner so that the POP/IMAP clients can deal with the messsages appropriately. For a small site with a few users (few 1000), using .qmail-(USER|default) or user-based implementation rules is fine. For an ISP with thousands of users, it's not good enough anymore. Even qmail-scanner-queue doesn't help protect servers from the constant deluge of malignant messages. I've been finding that at a small ISP (20k users), the final delivery is far too late in the process to deal with spam. Address harvesters (sending to 99% invalid addresses to find the 1% that don't bounce) and spam blasters (sending spam to 3 invalid recipients per message) tax the server processing hard enough to cause problems, particularly from bounce addresses to forged senders. As the spammers get more persistent or desperate, they've been less gracious about how they spam. In one case recently, I had a DDoS from 40 sites sending similar spam all at once to/through our server to thousands of bad addresses. Our servers spent a whole weekend trying to deliver the bounce messages until I could clean/drain the queues of 85000 bounce messages. There's not much that qmail-scanner can do itself to protect the server. I am using two tools for the benefit of my users: SpamAssassin (www.spamassassin.org) Vexira virus scanner (www.centralcommand.com) If the spammers weren't too peristent, I'd be able to just use qmail-scanner-queue.pl and be mostly done. This worked for a couple months before our ISP became a heavily hit target (60% spam, 25% malignant spam). My implementation now includes: a qmail-smtpd that rejects mail based on environment variables set from tcp.smtp. a qmail.c hacked to provde better SMTP error codes based on qq exit codes. a rewitten qmail-scanner-queue that is highly optimized at letting SA/spamc and Vexira do their job with minimal system resources a qmail-send that injects bounce messages to the sender only when it's a non-malignant message (one-to-one communication to a valid recipient) a qmail-send that puts messages into a holding queue rather than fully processing them right away. An asynchronous program comes by and processes each message in the holding queue linearly to prevent load swings from simultaneous qmail-send/vdelivermail instances. a procmail-like perl program responsible for final delivery that queries a mysql database for a user's spam preference and uses those preferences to tag/delete/pass messages based on SA scores and user-defined keywords. A coworker made a web user interface. added functionality to auto-add and auto-remove statistically defined address harvesters and spam blasters to my tcp.smtp block lists (with appropriate 400 or 500 messages based on severity) a program to create a cdb database of valid users to help the filtering programs detect how many valid vs invalid users an IP address or netblock is attempting to send to. This my the third round of an on-and-off 6-month long fight. It's not about filtering spam anymore, it's about protecting our mail servers. As I leave, I have a big I told you so about how our CEO should have just subscribed to BigFish/Frontbridge and paying the extra money instead of going it alone. It would have saved money and reduced downtime if SPAM processing weren't our problem. The system is complex (some new perl/SspeedyCGI programs plus several patches to qmail and one to vdelivermail), but it actively provides negative feedback to spammers and harvesters with (hopefully) little to no administration from a mail administrator. The good news is that I'm about to finish up, and I don't have any IP restricions with the ISP, so I believe I'll be able to share most of my work. I hope to be posting some patches and programs soon. Another approach could have been to just integrate everything into SpamAssassin, but it's getting too huge already. Each of thousands of 4K messages doen't need to go through a program that sucks 16MB RSS memory. A large program isn't the most efficient place to block/route messages. -- Eric Ziegast internet!vix.com!ziegast Winning another battle in the losing war against spam.
