Re: [dspam-dev] PHP UI alternative to dspamCC
Mark Rogers wrote: Jason Axley wrote: One basic feature I'd like to see is the ability to map user logins (e.g. from LDAP) to one or more identities that dspam knows about. How would you like this to work? Personally I'd like access from one login, but to still have the quarantines separate (eg separate IMAP folders). Primarily, I'm interested in the 1:1 cardinality mapping of login ID:dspam ID working to start. e.g. on my system I may have mail routed through for [EMAIL PROTECTED], but my LDAP login is jason. Also, I have other users with mail forwarding addresses such as [EMAIL PROTECTED] where the LDAP login might be sally. So, I'd want to configure: jason -> [EMAIL PROTECTED] sally -> [EMAIL PROTECTED] Then, the web UI would authorize these logins to access the corresponding dspam quarantine/queue (and remember, users who don't quarantine also need to use the web UI to retrain if you don't want to set up retraining aliases) Then, the more advanced setup would be 1:N cardinality of login ID:dspam IDs. I had to tortutously configure my mail system to get dspam to use a single user ID for retraining all of the myriad accounts that I have. So, this is not necessarily needed for me now, but would be a nice option to have. jason -> [EMAIL PROTECTED] jason -> [EMAIL PROTECTED] jason -> [EMAIL PROTECTED] jason -> [EMAIL PROTECTED] -Jason
Re: [dspam-dev] PHP UI alternative to dspamCC
Steve wrote: Original-Nachricht Datum: Fri, 08 Feb 2008 09:47:12 + Von: Mark Rogers <[EMAIL PROTECTED]> An: dspam-dev@lists.nuclearelephant.com Betreff: Re: [dspam-dev] PHP UI alternative to dspamCC Steve wrote: It depends how you retrain. If you have the tokens already inside the storage engine and linked to a signature, then retraining with just the signature is enough. If you don't store the tokens with a signature inside the storage engine or you purged the signature including the tokens already from the storage engine, then training with just the signature is not the way to go. It would be signature based that I'm interested in. As I understand it I do have to pass a user to dspam when I do the retraining (is that correct? I'm not at my server to check), This can be right and wrong. Right: When you have the user uid in the signature and you allow DSPAM to switch the user when retraining, then you don't need to specify a user while retraining. Sort-of-right: When you have the user uid in the signature and you allow DSPAM to switch the user when retraining, then you don't need to specify _different_ users when retraining. But, because the dspam command argument validation requires the user parameter, you must specify _something_ as the user, and that has to be a valid user on the system who dspam knows about. That's why lots of examples show using root or something like that in the generic retrain aliases. BTW, it would be nice to fix the dspam command to not barf if you don't specify a user argument if in dspam.conf you have the uidinsignature options set Very confusing, as this thread indicates. -Jason
Re: [dspam-dev] PHP UI alternative to dspamCC
Original-Nachricht > Datum: Fri, 08 Feb 2008 09:47:12 + > Von: Mark Rogers <[EMAIL PROTECTED]> > An: dspam-dev@lists.nuclearelephant.com > Betreff: Re: [dspam-dev] PHP UI alternative to dspamCC > Steve wrote: > > It depends how you retrain. If you have the tokens already inside the > storage engine and linked to a signature, then retraining with just the > signature is enough. If you don't store the tokens with a signature inside the > storage engine or you purged the signature including the tokens already from > the storage engine, then training with just the signature is not the way > to go. > > > > It would be signature based that I'm interested in. > > As I understand it I do have to pass a user to dspam when I do the > retraining (is that correct? I'm not at my server to check), > This can be right and wrong. Right: When you have the user uid in the signature and you allow DSPAM to switch the user when retraining, then you don't need to specify a user while retraining. Wrong: If you have plain signatures without the uid of the user in it, then you must specify the user name whit the retraining command. > so does it > basically not matter what user I give dspam if I'm giving it a signature? > No. It does matter. See above. > -- > Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555 > Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG Steve -- GMX FreeMail: 1 GB Postfach, 5 E-Mail-Adressen, 10 Free SMS. Alle Infos und kostenlose Anmeldung: http://www.gmx.net/de/go/freemail
Re: [dspam-dev] PHP UI alternative to dspamCC
Steve wrote: It depends how you retrain. If you have the tokens already inside the storage engine and linked to a signature, then retraining with just the signature is enough. If you don't store the tokens with a signature inside the storage engine or you purged the signature including the tokens already from the storage engine, then training with just the signature is not the way to go. It would be signature based that I'm interested in. As I understand it I do have to pass a user to dspam when I do the retraining (is that correct? I'm not at my server to check), so does it basically not matter what user I give dspam if I'm giving it a signature? -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555 Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG
Re: [dspam-dev] PHP UI alternative to dspamCC
Original-Nachricht > Datum: Thu, 07 Feb 2008 08:54:12 + > Von: Mark Rogers <[EMAIL PROTECTED]> > An: dspam-dev@lists.nuclearelephant.com > CC: Jason Axley <[EMAIL PROTECTED]> > Betreff: Re: [dspam-dev] PHP UI alternative to dspamCC > Jason Axley wrote: > > One basic feature I'd like to see is the ability to map user logins > > (e.g. from LDAP) to one or more identities that dspam knows about. > > How would you like this to work? Personally I'd like access from one > login, but to still have the quarantines separate (eg separate IMAP > folders). > > One thing I've never been 100% sure about: when retraining does dspam > actually need to know the recipient, or is just the signature sufficient? > It depends how you retrain. If you have the tokens already inside the storage engine and linked to a signature, then retraining with just the signature is enough. If you don't store the tokens with a signature inside the storage engine or you purged the signature including the tokens already from the storage engine, then training with just the signature is not the way to go. > > e.g. login jason can access [EMAIL PROTECTED], [EMAIL PROTECTED], etc. > > Would you be training dspam separately for the different addresses or > share the training between them? > > -- > Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555 > Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG -- Pt! Schon vom neuen GMX MultiMessenger gehört? Der kann`s mit allen: http://www.gmx.net/de/go/multimessenger
Re: [dspam-dev] PHP UI alternative to dspamCC
Jason Axley wrote: One basic feature I'd like to see is the ability to map user logins (e.g. from LDAP) to one or more identities that dspam knows about. How would you like this to work? Personally I'd like access from one login, but to still have the quarantines separate (eg separate IMAP folders). One thing I've never been 100% sure about: when retraining does dspam actually need to know the recipient, or is just the signature sufficient? e.g. login jason can access [EMAIL PROTECTED], [EMAIL PROTECTED], etc. Would you be training dspam separately for the different addresses or share the training between them? -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555 Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG
Re: [dspam-dev] PHP UI alternative to dspamCC
One basic feature I'd like to see is the ability to map user logins (e.g. from LDAP) to one or more identities that dspam knows about. The current CGI UI assumes that your login == your dspam user ID, which is not the case. e.g. I have a single mail server serving multiple domains and unless I create logins in LDAP that match the full email address, nobody can log in to manage their quarantine or retrain from the UI! e.g. login jason can access [EMAIL PROTECTED], [EMAIL PROTECTED], etc. -Jason
Re: [dspam-dev] PHP UI alternative to dspamCC
Alex Tomlins wrote: Ok, I had a look at doing Maildir delivery, and it can be done quite easily without modifying the dspam code. We just need to specify a QuarantineAgent in dspam.conf. I've knocked together a quick perl script that will do the trick. See attached. you can use it by specifying QuarantineAgent as below: QuarantineAgent "/var/spool/dspam/dspam-quarantine.pl %u" That all sounds promising, thanks for looking into it. At the moment it assumes dspam is configured for Domain Scale. It also assumes that a users data directory exists before it is called - I'm not 100% sure that this is a valid assumption. Those are just "details" :-) In the meantime, I've been looking at the log handling and have written a PHP script to cache logs into a MySQL database, which should make a huge difference to the time taken to process the history page and other reports. I only wish I had more time to work on this, I think there's a lot of good that can be done here. -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555 Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG
RE: [dspam-dev] PHP UI alternative to dspamCC
For example in big systems you only need to store that message itself only once. It save space and resources. > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf > Of Mark Rogers > Sent: Tuesday, January 29, 2008 5:22 PM > To: dspam-dev@lists.nuclearelephant.com > Subject: Re: [dspam-dev] PHP UI alternative to dspamCC > > Jani Partanen wrote: > > Database quarantine is something I really would like to see. > > For what purpose? > > I thought about this from a speed perspective (in particular > when sorting by criteria such as confidence levels). However, > simple text indexes achieves this pretty well without the > database dependency. > > Is there another benefit of moving to a database I hadn't > thought about? > As mentioned elsewhere, moving the logs to a database would > suit me a lot, so I'm looking for good excuses! > > On the basis of "picking the best tool for the job" and "not > reinventing the wheel" either a database or an IMAP server > would fit; suitability for handling email specifically might > suggest IMAP over database though, not least the fact that > (if necessary) a mail client could be configured to talk to > IMAP pretty easily, eg for sending back missed spam. > > Alex Tomlins wrote: > > I'd second that... > > > > I'm not sure about the IMAP idea. Something I like about > dspam at the > > moment is that it is a self-contained solution (that's most > of why I > > moved to it from spamassassin). If you start feeding stuff out to > > IMAP servers etc, you end up with more external dependencies. > > This seems to be a good argument against IMAP servers *AND* > database servers. > > If anything, it's more likely that a typical mail server will > have an IMAP server already present than a database. > > -- > Mark Rogers // More Solutions Ltd (Peterborough Office) // > 0845 45 89 555 Registered in England (0456 0902) at 13 Clarke > Rd, Milton Keynes, MK1 1LG > > >
Re: [dspam-dev] PHP UI alternative to dspamCC
Ok, I had a look at doing Maildir delivery, and it can be done quite easily without modifying the dspam code. We just need to specify a QuarantineAgent in dspam.conf. I've knocked together a quick perl script that will do the trick. See attached. you can use it by specifying QuarantineAgent as below: QuarantineAgent "/var/spool/dspam/dspam-quarantine.pl %u" At the moment it assumes dspam is configured for Domain Scale. It also assumes that a users data directory exists before it is called - I'm not 100% sure that this is a valid assumption. any thoughts? Alex Alex Tomlins wrote: > Forgot to copy the list See below. > > Alex Tomlins wrote: >> >> >> Mark Rogers wrote: >>> Alex Tomlins wrote: By database I was thinking whatever database backend dspam was using (hash, mysql, postgres etc.), not specifically a RDBMS. While implementing this it would make sense to put the user logs in there as well. The advantage to doing this is that all the data for dspam would be in one place as opposed to the current setup where some of it is in a database, and some of it is on the filesystem. >>> >>> OK, I see your point. However, I'm not sure suited how well a hash >>> database would be for storing quarantine emails - or, for that >>> matter, even a RDBMS. Logging to a RDBMS would be helpful given how >>> much processing we do of dspam log files, but in most cases that can >>> be improved simply through better ways of handling the text files. >>> >>> I'm strongly of the opinion that mbox is horrible, but which >>> alternative is best I'm open to suggestion on. >> I agree with that sentiment.. Personally I'd just replace it with a >> Maildir. This should require minimal changes to dspam (and could >> even be added as a config option). It also has the bonus that you >> can clear out old messages just using the mtime (which can be done >> with a single find command). It might also be possible to use >> symlinks so that the maildir is actually a subfolder of a users >> mailbox, and can then be accessed via IMAP (I know Courier-imap is >> fine with adding messages to other folders as well as the inbox). >> >> If there is a better option for the log files, then it should be fine >> to leave them where they are. >> >> Actually, using maildir would enable much more fine grained expunging >> methods, where you could for example expunge all spam over 90% >> confidence after 7 days, over 75% in 14 days, 60% in 30 days etc... >> >> I like that idea. I might even have a look at the code for that. >> Now that DJB has put qmail in the public domain it should be simple >> enough to pull the Maildir delivery code out of there. >> >> Alex >> > -- Alex Tomlins Email/Jabber: [EMAIL PROTECTED] There are two kinds of people in the world: those who finish what they started dspam-quarantine.pl Description: Perl program
Re: [dspam-dev] PHP UI alternative to dspamCC
On Tue, Jan 29, 2008 at 03:49:10PM +, Mark Rogers wrote: > Alex Tomlins wrote: >> By database I was thinking whatever database backend dspam was using >> (hash, mysql, postgres etc.), not specifically a RDBMS. While >> implementing this it would make sense to put the user logs in there as >> well. The advantage to doing this is that all the data for dspam would be >> in one place as opposed to the current setup where some of it is in a >> database, and some of it is on the filesystem. >> > > OK, I see your point. However, I'm not sure suited how well a hash database > would be for storing quarantine emails - or, for that matter, even a RDBMS. > Logging to a RDBMS would be helpful given how much processing we do of > dspam log files, but in most cases that can be improved simply through > better ways of handling the text files. > > I'm strongly of the opinion that mbox is horrible, but which alternative is > best I'm open to suggestion on. > Also, storing files in an SQL database has a much, much higher system resource footprint then using the filesystem, which is already optimized for file storage. I concur that mbox is a poor alternative, but a maildir option should be suitable. Ken
Re: [dspam-dev] PHP UI alternative to dspamCC
Forgot to copy the list See below. Alex Tomlins wrote: Mark Rogers wrote: Alex Tomlins wrote: By database I was thinking whatever database backend dspam was using (hash, mysql, postgres etc.), not specifically a RDBMS. While implementing this it would make sense to put the user logs in there as well. The advantage to doing this is that all the data for dspam would be in one place as opposed to the current setup where some of it is in a database, and some of it is on the filesystem. OK, I see your point. However, I'm not sure suited how well a hash database would be for storing quarantine emails - or, for that matter, even a RDBMS. Logging to a RDBMS would be helpful given how much processing we do of dspam log files, but in most cases that can be improved simply through better ways of handling the text files. I'm strongly of the opinion that mbox is horrible, but which alternative is best I'm open to suggestion on. I agree with that sentiment.. Personally I'd just replace it with a Maildir. This should require minimal changes to dspam (and could even be added as a config option). It also has the bonus that you can clear out old messages just using the mtime (which can be done with a single find command). It might also be possible to use symlinks so that the maildir is actually a subfolder of a users mailbox, and can then be accessed via IMAP (I know Courier-imap is fine with adding messages to other folders as well as the inbox). If there is a better option for the log files, then it should be fine to leave them where they are. Actually, using maildir would enable much more fine grained expunging methods, where you could for example expunge all spam over 90% confidence after 7 days, over 75% in 14 days, 60% in 30 days etc... I like that idea. I might even have a look at the code for that. Now that DJB has put qmail in the public domain it should be simple enough to pull the Maildir delivery code out of there. Alex -- Alex Tomlins Email/Jabber: [EMAIL PROTECTED] There are two kinds of people in the world: those who finish what they started.
Re: [dspam-dev] PHP UI alternative to dspamCC
Alex Tomlins wrote: By database I was thinking whatever database backend dspam was using (hash, mysql, postgres etc.), not specifically a RDBMS. While implementing this it would make sense to put the user logs in there as well. The advantage to doing this is that all the data for dspam would be in one place as opposed to the current setup where some of it is in a database, and some of it is on the filesystem. OK, I see your point. However, I'm not sure suited how well a hash database would be for storing quarantine emails - or, for that matter, even a RDBMS. Logging to a RDBMS would be helpful given how much processing we do of dspam log files, but in most cases that can be improved simply through better ways of handling the text files. I'm strongly of the opinion that mbox is horrible, but which alternative is best I'm open to suggestion on. -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555 Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG
Re: [dspam-dev] PHP UI alternative to dspamCC
Mark Rogers wrote: Alex Tomlins wrote: I'd second that... I'm not sure about the IMAP idea. Something I like about dspam at the moment is that it is a self-contained solution (that's most of why I moved to it from spamassassin). If you start feeding stuff out to IMAP servers etc, you end up with more external dependencies. This seems to be a good argument against IMAP servers *AND* database servers. If anything, it's more likely that a typical mail server will have an IMAP server already present than a database. By database I was thinking whatever database backend dspam was using (hash, mysql, postgres etc.), not specifically a RDBMS. While implementing this it would make sense to put the user logs in there as well. The advantage to doing this is that all the data for dspam would be in one place as opposed to the current setup where some of it is in a database, and some of it is on the filesystem. Alex -- Alex Tomlins Email/Jabber: [EMAIL PROTECTED] There are two kinds of people in the world: those who finish what they started.
Re: [dspam-dev] PHP UI alternative to dspamCC
Jani Partanen wrote: Database quarantine is something I really would like to see. For what purpose? I thought about this from a speed perspective (in particular when sorting by criteria such as confidence levels). However, simple text indexes achieves this pretty well without the database dependency. Is there another benefit of moving to a database I hadn't thought about? As mentioned elsewhere, moving the logs to a database would suit me a lot, so I'm looking for good excuses! On the basis of "picking the best tool for the job" and "not reinventing the wheel" either a database or an IMAP server would fit; suitability for handling email specifically might suggest IMAP over database though, not least the fact that (if necessary) a mail client could be configured to talk to IMAP pretty easily, eg for sending back missed spam. Alex Tomlins wrote: I'd second that... I'm not sure about the IMAP idea. Something I like about dspam at the moment is that it is a self-contained solution (that's most of why I moved to it from spamassassin). If you start feeding stuff out to IMAP servers etc, you end up with more external dependencies. This seems to be a good argument against IMAP servers *AND* database servers. If anything, it's more likely that a typical mail server will have an IMAP server already present than a database. -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555 Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG
Re: [dspam-dev] PHP UI alternative to dspamCC
Bolla Péter wrote: PS: *well, if you backup dspam spool. You should, imho. One of the problems with spam is that it can create a massive system overhead that causes other problems. I would be concerned about the scalability of backing up the quarantine spool in the event that a massive spam flood arrived. Something intelligent that backed up anything less than (say) 70% confidence would be nice though. Now I've started thinking about the IMAP options, different confidence levels could be filtered at that point giving a low confidence IMAP box to backup, and a higher confidence one that's pruned more often and generally less well "looked after". -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555 Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG
Re: [dspam-dev] PHP UI alternative to dspamCC
Bolla Péter wrote: Also I would oppose any server specific solution, IMAP is a quite well followed standard, afaik. Absolutely agreed on that one. (So both read and write the mails throug IMAP, without any direct file access. Some IMAP servers don't even let you access the files directly.) Worth knowing, Seems odd; I thought most MTAs would expect to drop into Maildir directly rather than via an IMAP server. In any case I'll keep all the parts separate. The web UI, if I go down that route, will just talk IMAP; getting the mail from quarantine into it will be separate. Kenneth Marshall wrote: One of the advantages of the current DSPAM quarantine is that the ham/spam messages are not delivered to the production mail system unless released by the user. [...] I guess that one idea would be to have a separate IMAP server for the DSPAM UI. Certainly the way I see it, whether or not IMAP is used for the quarantine, the quarantine will still be a separate place from the normal mailbox. IMAP would just be a standard interface into it. Regarding the performance of the UI. The activity plots are nice, but they parse the entire logfile each time they are built. It would be nice to cache just the information for the display for historical data to avoid the need to re-read the logfile over and over. This would allow us to keep the daily activity for a longer period without impacting the display update time. The log files are a particular area of interest for me for this reason, but I've not found a good way to improve them yet. The obvious thing to me would be to drop logs into a database using something like rsyslog, but that's too big a change to expect sysadmins to do just for installing a web UI. -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555 Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG
Re: [dspam-dev] PHP UI alternative to dspamCC
I'd second that... I'm not sure about the IMAP idea. Something I like about dspam at the moment is that it is a self-contained solution (that's most of why I moved to it from spamassassin). If you start feeding stuff out to IMAP servers etc, you end up with more external dependencies. And you can already do that in a way. If you set dspam to tag messages rather than quarantine them, you can then just have a filter in your mail server that then puts all tagged messages into a specific folder as opposed to the inbox. thanks, Alex Jani Partanen wrote: Database quarantine is something I really would like to see. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Kenneth Marshall Sent: Tuesday, January 29, 2008 4:36 PM To: Bolla P?ter Cc: dspam-dev@lists.nuclearelephant.com Subject: Re: [dspam-dev] PHP UI alternative to dspamCC One of the advantages of the current DSPAM quarantine is that the ham/spam messages are not delivered to the production mail system unless released by the user. Our IMAP system keeps any message for 8 days, before it is removed and if quarantine messages were stored there, they would end up consuming disk resources and backup resources far longer than the current setup. I guess that one idea would be to have a separate IMAP server for the DSPAM UI. Regarding the performance of the UI. The activity plots are nice, but they parse the entire logfile each time they are built. It would be nice to cache just the information for the display for historical data to avoid the need to re-read the logfile over and over. This would allow us to keep the daily activity for a longer period without impacting the display update time. Some ideas. Ken On Tue, Jan 29, 2008 at 08:29:11AM -0600, Bolla P?ter wrote: Well, in my idea DSPAM would have been converted to use IMAP too, instead of directly manipulating an mbox. Regarding imap server, I would not recommend anythig, as I only know cyrus, what I use. But I think that in the majority of dspam setups there is an imap server already in the system somewhere, so that would be not much of a performance/maintenance issue. Also I would oppose any server specific solution, IMAP is a quite well followed standard, afaik. (So both read and write the mails throug IMAP, without any direct file access. Some IMAP servers don't even let you access the files directly.) Peter Mark Rogers wrote: Mark Rogers wrote: Of-course mbox -> IMAP scripts probably exist, so I could still look at this option now. Now that my brain has started... Of-course I don't need to parse the mbox file and pass the mails to an IMAP server, I just need to convert them to Maildir format and let a normal IMAP server pick them up from there. Which makes things far simpler! Doh! -- Alex Tomlins Email/Jabber: [EMAIL PROTECTED] There are two kinds of people in the world: those who finish what they started.
RE: [dspam-dev] PHP UI alternative to dspamCC
Database quarantine is something I really would like to see. > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf > Of Kenneth Marshall > Sent: Tuesday, January 29, 2008 4:36 PM > To: Bolla P?ter > Cc: dspam-dev@lists.nuclearelephant.com > Subject: Re: [dspam-dev] PHP UI alternative to dspamCC > > One of the advantages of the current DSPAM quarantine is that > the ham/spam messages are not delivered to the production > mail system unless released by the user. Our IMAP system > keeps any message for 8 days, before it is removed and if > quarantine messages were stored there, they would end up > consuming disk resources and backup resources far longer than > the current setup. I guess that one idea would be to have a > separate IMAP server for the DSPAM UI. > > Regarding the performance of the UI. The activity plots are > nice, but they parse the entire logfile each time they are > built. It would be nice to cache just the information for the > display for historical data to avoid the need to re-read the > logfile over and over. This would allow us to keep the daily > activity for a longer period without impacting the display > update time. > > Some ideas. > Ken > > On Tue, Jan 29, 2008 at 08:29:11AM -0600, Bolla P?ter wrote: > > Well, in my idea DSPAM would have been converted to use IMAP too, > > instead of directly manipulating an mbox. > > > > Regarding imap server, I would not recommend anythig, as I > only know > > cyrus, what I use. But I think that in the majority of dspam setups > > there is an imap server already in the system somewhere, so > that would > > be not much of a performance/maintenance issue. Also I would oppose > > any server specific solution, IMAP is a quite well followed > standard, > > afaik. (So both read and write the mails throug IMAP, without any > > direct file access. Some IMAP servers don't even let you access the > > files directly.) > > > > Peter > > > > Mark Rogers wrote: > >> Mark Rogers wrote: > >>> Of-course mbox -> IMAP scripts probably exist, so I could > still look > >>> at this option now. > >> Now that my brain has started... > >> Of-course I don't need to parse the mbox file and pass the > mails to > >> an IMAP server, I just need to convert them to Maildir > format and let > >> a normal IMAP server pick them up from there. Which makes > things far > >> simpler! > >> Doh! > > > >
Re: [dspam-dev] PHP UI alternative to dspamCC
Hey, Kenneth Marshall wrote: [...] unless released by the user. Our IMAP system keeps any message for 8 days, before it is removed and if quarantine messages were stored there, they would end up consuming disk resources and backup resources far longer than the current setup. I guess that one idea [...] Well, in the current case the the quarantain still consumes disk and backup* resources, it's just more difficult to handle :) Or you mean the quarantain holds messages for 8 days? Well, old messages could be purged out of the IMAP quarantain foders too, every day by a short script. Peter PS: *well, if you backup dspam spool. You should, imho.
Re: [dspam-dev] PHP UI alternative to dspamCC
One of the advantages of the current DSPAM quarantine is that the ham/spam messages are not delivered to the production mail system unless released by the user. Our IMAP system keeps any message for 8 days, before it is removed and if quarantine messages were stored there, they would end up consuming disk resources and backup resources far longer than the current setup. I guess that one idea would be to have a separate IMAP server for the DSPAM UI. Regarding the performance of the UI. The activity plots are nice, but they parse the entire logfile each time they are built. It would be nice to cache just the information for the display for historical data to avoid the need to re-read the logfile over and over. This would allow us to keep the daily activity for a longer period without impacting the display update time. Some ideas. Ken On Tue, Jan 29, 2008 at 08:29:11AM -0600, Bolla P?ter wrote: > Well, in my idea DSPAM would have been converted to use IMAP too, instead > of directly manipulating an mbox. > > Regarding imap server, I would not recommend anythig, as I only know cyrus, > what I use. But I think that in the majority of dspam setups there is an > imap server already in the system somewhere, so that would be not much of a > performance/maintenance issue. Also I would oppose any server specific > solution, IMAP is a quite well followed standard, afaik. (So both read and > write the mails throug IMAP, without any direct file access. Some IMAP > servers don't even let you access the files directly.) > > Peter > > Mark Rogers wrote: >> Mark Rogers wrote: >>> Of-course mbox -> IMAP scripts probably exist, so I could still look at >>> this option now. >> Now that my brain has started... >> Of-course I don't need to parse the mbox file and pass the mails to an >> IMAP server, I just need to convert them to Maildir format and let a >> normal IMAP server pick them up from there. Which makes things far >> simpler! >> Doh! >
Re: [dspam-dev] PHP UI alternative to dspamCC
Well, in my idea DSPAM would have been converted to use IMAP too, instead of directly manipulating an mbox. Regarding imap server, I would not recommend anythig, as I only know cyrus, what I use. But I think that in the majority of dspam setups there is an imap server already in the system somewhere, so that would be not much of a performance/maintenance issue. Also I would oppose any server specific solution, IMAP is a quite well followed standard, afaik. (So both read and write the mails throug IMAP, without any direct file access. Some IMAP servers don't even let you access the files directly.) Peter Mark Rogers wrote: Mark Rogers wrote: Of-course mbox -> IMAP scripts probably exist, so I could still look at this option now. Now that my brain has started... Of-course I don't need to parse the mbox file and pass the mails to an IMAP server, I just need to convert them to Maildir format and let a normal IMAP server pick them up from there. Which makes things far simpler! Doh!
Re: [dspam-dev] PHP UI alternative to dspamCC
Mark Rogers wrote: Of-course mbox -> IMAP scripts probably exist, so I could still look at this option now. Now that my brain has started... Of-course I don't need to parse the mbox file and pass the mails to an IMAP server, I just need to convert them to Maildir format and let a normal IMAP server pick them up from there. Which makes things far simpler! Doh! -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555 Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG
Re: [dspam-dev] PHP UI alternative to dspamCC
Bolla Péter wrote: Why to bother writing mail storage system, while it has been already done by others, and maintained well in several real good IMAP servers, and IMAP client libraries (libs both for perl and php)? I think many-many hours of development AND maintenance time could be spared if dspam would just connect to an IMAP server with it's own user, and store quarantained mails for separate users in separate IMAP folders. I think you're right on that one. For the time being I'll push ahead with the mbox variety as it's there and I need a better way to work with it than the existing web UI, but building it around IMAP would be better in the long run. Of-course mbox -> IMAP scripts probably exist, so I could still look at this option now. (Instead of building mbox index, move content of mbox into IMAP and continue from there.) I've not done much with IMAP, but if people think this is the right route for now (with dspam changing in the medium term to drop straight into IMAP) I'll look into it further. Further advantage is that with an advanced imap server, which handles ACLs, you could add the proper access rights to the qurantain folder for its respective user, so you could manage your quarantain right from your mail reader. How's that? What IMAP server would you recommend? One thing I'd note is that an existing dspam installation has pretty low system requirements, I wouldn't want to see that jump significantly. The existing web UI suffers from trying to do too much in RAM now (my PHP version does much less that way), so the overhead of dropping spam into an IMAP server would be a consideration. My own dspam server is using Courier, btw. -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555 Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG
Re: [dspam-dev] PHP UI alternative to dspamCC
Hey, Everything sounds pretty good but another thing to consider might be modifying the existing dspam code to store messages in Maildir, instead of mbox? Or to store the entire quarantine (and the other WebUI files) inside of the database. Why to bother writing mail storage system, while it has been already done by others, and maintained well in several real good IMAP servers, and IMAP client libraries (libs both for perl and php)? I think many-many hours of development AND maintenance time could be spared if dspam would just connect to an IMAP server with it's own user, and store quarantained mails for separate users in separate IMAP folders. Further advantage is that with an advanced imap server, which handles ACLs, you could add the proper access rights to the qurantain folder for its respective user, so you could manage your quarantain right from your mail reader. How's that? My 2c. Peter
Re: [dspam-dev] PHP UI alternative to dspamCC
Kyle Johnson wrote: Sweet! It's good to see someone finally doing this. Foxdie in the #dspam channel has been working on a PHP conversion as well, but there has not been any progress as of late. You might want to send him a line. You got a name for him? Foxdie means nothing to me, sorry! I spoke to a couple of people off-list who'd been working on a PHP interface, but had run out of steam on it. I've got their code but only glanced at it so far, having started from scratch so far. Everything sounds pretty good but another thing to consider might be modifying the existing dspam code to store messages in Maildir, instead of mbox? Or to store the entire quarantine (and the other WebUI files) inside of the database. It crossed my mind, but I didn't want to impact anything I didn't need to. Not least so the existing UI doesn't completely break while I work on this. About the index idea - it should work. Dspam doesn't touch the WebUI, aside from appending quarantine files to it. Cool. It does make a big difference from the UI point of view (no surprise there!). Dov Zamir wrote: Firstly, I congratulate the initiative! Let's see what you say of the results :-) Secondly, I would like to see some thought on i18n in the interface. I'm happy to work with someone on that one, but as one of those terrible people who speak English as a first and only language it's not an area I know much about. I did at least think about separating language text from code, although at the moment a lot of what I have is pretty messy code to test some ideas. Good luck with the project. Please keep the list posted, perhaps some of us can find time to help out, if we know what's going on. Will do. Assuming I get anywhere with it, the results will be GPL in line with the rest of dSpam anyway. -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555 Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG
Re: [dspam-dev] PHP UI alternative to dspamCC
Mark Rogers wrote: I've started writing a PHP alternative to the existing dspamCC web interface. My primary objectives are to fix some things that annoy me personally about the existing one (the very slow loading/processing of large quarantine boxes, and the inability to sort/filter on history page, being the main ones). I know PHP far better than Perl otherwise I'd just modify the existing one, but having looked at the Perl code I think I'm better starting from scratch. What I wanted advice on is some strategy decisions I'm taking early on. Firstly: From a security point of view I'm writing a helper PHP script (not located under document root) which manipulates files as the dspam user. I'm using sudo and the sudoers file to control access to it. What's the general consensus on that method? I prefer it as it means all of the scripts processed through Apache still have no direct access to any of dspam's files, but I'm not a security expert. Secondly: I've written code to index the quarantine box, generating a user.mbox.index file in the dspam data directory alongside the user.mbox file. This (after the initial index build) does seem to speed the loading of the quarantine page significantly. Maybe I'm the only one who has users only check their quarantine when it reached 20k messages though. Anyway, can anyone see problems with this approach? Thirdly: When it comes to removing messages from the quarantine, do I have to take into account anything other than my own code? I assume dspam itself only ever appends mails to the quarantine mailbox, so beyond that what I do with it is up to me? Eg: I could mark messages for deletion in the index (rather than remove them from the .mbox file) and process them later, but would that cause problems for anything else? Finally: Are there any general thoughts on this project? Any demand from other users, willing helpers/testers, etc? At this point I can't make any promises as to how much time I'm going to be able to put in to it, so I don't want to get false hopes up, but I'm still interested to gauge interest. Firstly, I congratulate the initiative! Secondly, I would like to see some thought on i18n in the interface. At present I personally translate the interface to Hebrew by going into the source code and changing the original strings. Two problems with this are 1) It has to be repeated with every GUI update and 2) There is no option for any language other than the hard coded one (meaning Hebrew in my case). Good luck with the project. Please keep the list posted, perhaps some of us can find time to help out, if we know what's going on. This mail was sent via Mail-SeCure System. This footnote confirms that this email message has been scanned by PineApp Mail-SeCure for the presence of malicious code, vandals & computer viruses.
Re: [dspam-dev] PHP UI alternative to dspamCC
Mark Rogers wrote: I've started writing a PHP alternative to the existing dspamCC web interface. My primary objectives are to fix some things that annoy me personally about the existing one (the very slow loading/processing of large quarantine boxes, and the inability to sort/filter on history page, being the main ones). I know PHP far better than Perl otherwise I'd just modify the existing one, but having looked at the Perl code I think I'm better starting from scratch. What I wanted advice on is some strategy decisions I'm taking early on. Firstly: From a security point of view I'm writing a helper PHP script (not located under document root) which manipulates files as the dspam user. I'm using sudo and the sudoers file to control access to it. What's the general consensus on that method? I prefer it as it means all of the scripts processed through Apache still have no direct access to any of dspam's files, but I'm not a security expert. Secondly: I've written code to index the quarantine box, generating a user.mbox.index file in the dspam data directory alongside the user.mbox file. This (after the initial index build) does seem to speed the loading of the quarantine page significantly. Maybe I'm the only one who has users only check their quarantine when it reached 20k messages though. Anyway, can anyone see problems with this approach? Thirdly: When it comes to removing messages from the quarantine, do I have to take into account anything other than my own code? I assume dspam itself only ever appends mails to the quarantine mailbox, so beyond that what I do with it is up to me? Eg: I could mark messages for deletion in the index (rather than remove them from the .mbox file) and process them later, but would that cause problems for anything else? Finally: Are there any general thoughts on this project? Any demand from other users, willing helpers/testers, etc? At this point I can't make any promises as to how much time I'm going to be able to put in to it, so I don't want to get false hopes up, but I'm still interested to gauge interest. Sweet! It's good to see someone finally doing this. Foxdie in the #dspam channel has been working on a PHP conversion as well, but there has not been any progress as of late. You might want to send him a line. Everything sounds pretty good but another thing to consider might be modifying the existing dspam code to store messages in Maildir, instead of mbox? Or to store the entire quarantine (and the other WebUI files) inside of the database. About the index idea - it should work. Dspam doesn't touch the WebUI, aside from appending quarantine files to it. --Kyle
