Re: was: Allowing IMAP users to train spam/ham is:simplify training of misclassified emails

2012-03-22 Thread Kevin A. McGrail
Before anyone rushes ahead and puts any time or money into this. I think it's worth establishing whether it makes any significant difference. It solves several real world problems that I'm aware of but I agree it's not going to hold up 3.4.0 or be a top priority for me. regards, KAM

Re: was: Allowing IMAP users to train spam/ham is:simplify training of misclassified emails

2012-03-22 Thread RW
On Thu, 22 Mar 2012 07:59:39 -0400 Kevin A. McGrail wrote: > Yes and no. What you have missed is that David F Skoll is a key > author of MIMEDefang. They also publish a great COTS solution for > email filtering called CanIT. So his plugin is part of the commercial > product. AFAIK his Bayes uses

Re: was: Allowing IMAP users to train spam/ham is:simplify training of misclassified emails

2012-03-22 Thread Kevin A. McGrail
Yes and no. What you have missed is that David F Skoll is a key author of MIMEDefang. They also publish a great COTS solution for email filtering called CanIT. So his plugin is part of the commercial product. However, his idea is very elegant on tokens is an elegant idea. To extract them, I pla

Re: was: Allowing IMAP users to train spam/ham is:simplify training of misclassified emails

2012-03-22 Thread David F. Skoll
On Thu, 22 Mar 2012 07:51:07 +0100 Per-Erik Persson wrote: > Since we are on the subject of adding "magic links" to email header to > make it easier for nontech staff to report spam. > I don't understand how to extract the tokinzed data needed to > represent the specific email. We have an entire

Re: was: Allowing IMAP users to train spam/ham is:simplify training of misclassified emails

2012-03-22 Thread Robert Schetterer
Am 22.03.2012 09:15, schrieb xTrade Assessory: > Robert Schetterer wrote: >>> >> >> however , i have a ham/spam transport learn mail address, >> nearly null users forwards something to it, no wonder >> the false positve rate is nearly null >> >> in fact , there are systems with webmail guis for cla

Re: was: Allowing IMAP users to train spam/ham is:simplify training of misclassified emails

2012-03-22 Thread Per-Erik Persson
On 03/22/2012 07:59 AM, Robert Schetterer wrote: > Am 22.03.2012 07:51, schrieb Per-Erik Persson: >> Since we are on the subject of adding "magic links" to email header to >> make it easier for nontech staff to report spam. >> I don't understand how to extract the tokinzed data needed to represent

Re: was: Allowing IMAP users to train spam/ham is:simplify training of misclassified emails

2012-03-22 Thread xTrade Assessory
Robert Schetterer wrote: >> > > however , i have a ham/spam transport learn mail address, > nearly null users forwards something to it, no wonder > the false positve rate is nearly null > > in fact , there are systems with webmail guis for classify > spam i.e aol, reality shows users dont use it

Re: was: Allowing IMAP users to train spam/ham is:simplify training of misclassified emails

2012-03-22 Thread Robert Schetterer
Am 22.03.2012 07:51, schrieb Per-Erik Persson: > Since we are on the subject of adding "magic links" to email header to > make it easier for nontech staff to report spam. > I don't understand how to extract the tokinzed data needed to represent > the specific email. > Have I missed some plugin that

was: Allowing IMAP users to train spam/ham is:simplify training of misclassified emails

2012-03-21 Thread Per-Erik Persson
Since we are on the subject of adding "magic links" to email header to make it easier for nontech staff to report spam. I don't understand how to extract the tokinzed data needed to represent the specific email. Have I missed some plugin that everyone else knows about? The rest of the problem seem

Re: Allowing IMAP users to train spam/ham

2012-03-21 Thread Benny Pedersen
Den 2012-03-21 13:38, Michael Scheidell skrev: so, what would you manually learn? using dspam then its not a problem, it only needs dspam signature internet > postfix > dspam > postfix > exchange relay transport now exchange have the dspam signature and can report back if its spam or ham, h

Re: Allowing IMAP users to train spam/ham

2012-03-21 Thread David F. Skoll
On Wed, 21 Mar 2012 10:41:31 -0400 Michael Scheidell wrote: > But, what do you do about an email that was forwarded to someone else? > And, that someone else has one of those silly anti-malware plugins > that surfs to every url in any inbound email? By default, our system won't allow training un

Re: Allowing IMAP users to train spam/ham

2012-03-21 Thread Kevin A. McGrail
On 3/21/2012 10:41 AM, Michael Scheidell wrote: On 3/21/12 9:57 AM, Kevin A. McGrail wrote: Very elegant IMO. I'd love to look at moving some of the framework to support this into SA. Any objections? Won't be anything quick but it's a really great idea. We thought about this once. add (ie

Re: Allowing IMAP users to train spam/ham

2012-03-21 Thread Michael Scheidell
On 3/21/12 9:57 AM, Kevin A. McGrail wrote: Very elegant IMO. I'd love to look at moving some of the framework to support this into SA. Any objections? Won't be anything quick but it's a really great idea. We thought about this once. add (ie: modify body of email) with 'report spam', 'blac

Re: Allowing IMAP users to train spam/ham

2012-03-21 Thread Kevin A. McGrail
On 3/21/2012 10:03 AM, David F. Skoll wrote: On Wed, 21 Mar 2012 09:57:33 -0400 "Kevin A. McGrail" wrote: [Storing Bayes tokens on the server and retrieving them when training] Very elegant IMO. I'd love to look at moving some of the framework to support this into SA. Any objections? Won't

Re: Allowing IMAP users to train spam/ham

2012-03-21 Thread David F. Skoll
On Wed, 21 Mar 2012 09:57:33 -0400 "Kevin A. McGrail" wrote: [Storing Bayes tokens on the server and retrieving them when training] > Very elegant IMO. I'd love to look at moving some of the framework > to support this into SA. Any objections? Won't be anything quick > but it's a really great

Re: Allowing IMAP users to train spam/ham

2012-03-21 Thread Kevin A. McGrail
On 3/21/2012 9:30 AM, David F. Skoll wrote: Actually, there's a third way and it's what we do (but difficult to set up with pure SpamAssassin.) We tokenize inbound messages and store the tokens on the server. In each message, we add links for doing training. When you click on a training link, t

Re: Allowing IMAP users to train spam/ham

2012-03-21 Thread David F. Skoll
On Wed, 21 Mar 2012 13:44:49 +0100 Matus UHLAR - fantomas wrote: > Mangling data by exchange is a big. problem when trying to filter > spam in front of it. I see two ways to avoid this problem: > - use spam server for exchange. We use one from GFI, with quite good > results. > - you can use spa

Re: Allowing IMAP users to train spam/ham

2012-03-21 Thread RW
On Wed, 21 Mar 2012 10:06:58 +0100 Matus UHLAR - fantomas wrote: > >On Fri, 9 Mar 2012 16:38:49 +0100 > >Matus UHLAR - fantomas wrote: > >No, it isn't. Bayes is a statistical filter it needs to learn a lot > >of diverse spam and ham to reach it's optimum accuracy. It's been > >demonstrated on Bo

Re: Allowing IMAP users to train spam/ham

2012-03-21 Thread Matus UHLAR - fantomas
On 3/21/12 5:06 AM, Matus UHLAR - fantomas wrote: there are two problems when requiring users to manually learn on everythhing. - it's more work to implement - it's more work for users to do the training. On 21.03.12 08:38, Michael Scheidell wrote: and, if 95% of the users are using microsoft

Re: Allowing IMAP users to train spam/ham

2012-03-21 Thread Michael Scheidell
On 3/21/12 5:06 AM, Matus UHLAR - fantomas wrote: there are two problems when requiring users to manually learn on everythhing. - it's more work to implement - it's more work for users to do the training. and, if 95% of the users are using microsoft exchange, exchange will horribly mangle the h

Re: Allowing IMAP users to train spam/ham

2012-03-21 Thread Matus UHLAR - fantomas
On Fri, 9 Mar 2012 16:38:49 +0100 Matus UHLAR - fantomas wrote: You can of course configure mailer to train automatically on anything received/delivered. However this would apparently cause much more FP's and FN's rate than letting user train only those that misfire. On 10.03.12 00:07, RW wrot

Re: Allowing IMAP users to train spam/ham

2012-03-11 Thread RW
On Sun, 11 Mar 2012 13:56:52 -0600 LuKreme wrote: > > On 09 Mar 2012, at 17:07 , RW wrote: > > > It's been demonstrated on Bogofilter that "train-on-everything" > > outperforms "train-on-error" on the same corpora. They both end-up > > with similar accuracy, but "train-on-everything" gets there

Re: Allowing IMAP users to train spam/ham

2012-03-11 Thread LuKreme
On 09 Mar 2012, at 17:07 , RW wrote: > It's been demonstrated on Bogofilter that "train-on-everything" outperforms > "train-on-error" on the same corpora. They both end-up with similar accuracy, > but "train-on-everything" gets there very much faster. But training is exceedingly slow. Under no

Re: Allowing IMAP users to train spam/ham

2012-03-09 Thread RW
On Fri, 9 Mar 2012 16:38:49 +0100 Matus UHLAR - fantomas wrote: > You can of course configure mailer to train automatically on anything > received/delivered. However this would apparently cause much more > FP's and FN's rate than letting user train only those that misfire. The use of the word

Re: Allowing IMAP users to train spam/ham

2012-03-09 Thread Matus UHLAR - fantomas
On Fri, 9 Mar 2012 08:38:21 +0100 Matus UHLAR - fantomas wrote: >> On 05.03.12 12:15, RW wrote: >> >I don't like it. It relies on FPs being removed from the SPAM >> >folder rather than spam being sent to a learn-spam folder. >On Wed, 7 Mar 2012 15:35:05 +0100 >Matus UHLAR - fantomas wrote: >> P

Re: Allowing IMAP users to train spam/ham

2012-03-09 Thread RW
On Fri, 9 Mar 2012 08:38:21 +0100 Matus UHLAR - fantomas wrote: > >> On 05.03.12 12:15, RW wrote: > >> >I don't like it. It relies on FPs being removed from the SPAM > >> >folder rather than spam being sent to a learn-spam folder. > > >On Wed, 7 Mar 2012 15:35:05 +0100 > >Matus UHLAR - fantomas w

Re: Allowing IMAP users to train spam/ham

2012-03-08 Thread Matus UHLAR - fantomas
On 05.03.12 12:15, RW wrote: >I don't like it. It relies on FPs being removed from the SPAM folder >rather than spam being sent to a learn-spam folder. On Wed, 7 Mar 2012 15:35:05 +0100 Matus UHLAR - fantomas wrote: Pardon me, but: Usage for end users *move mail into SPAM folder to clas

Re: Allowing IMAP users to train spam/ham

2012-03-07 Thread RW
On Wed, 7 Mar 2012 15:35:05 +0100 Matus UHLAR - fantomas wrote: > On 05.03.12 12:15, RW wrote: > >I don't like it. It relies on FPs being removed from the SPAM folder > >rather than spam being sent to a learn-spam folder. > > Pardon me, but: > > Usage for end users > > *move mail into SPA

Re: Allowing IMAP users to train spam/ham

2012-03-07 Thread Matus UHLAR - fantomas
On 04.03.12 14:02, RW wrote: >An alternative would be to be more selective. I'm not sure if this is >specific to dovecot but when I copy/move a file in IMAP the new >maildir file has the same mtime, but a new epoch time in the file >name. What you might do is generate a list of filenames that co

Re: Allowing IMAP users to train spam/ham

2012-03-06 Thread Christian Grunfeld
Hi, do you have per virtual user Bayes training? or sitewide virtual user? Because I have a setup like yours and everything goes fine ! In my setup users move by hand to spam folder FNs and retrieve from spam folder to inbox FPs ! When they make that movements a script copies those spam/ham to a s

Re: Allowing IMAP users to train spam/ham

2012-03-05 Thread Robert Schetterer
Am 05.03.2012 13:15, schrieb RW: > On Mon, 5 Mar 2012 10:54:22 +0100 > Matus UHLAR - fantomas wrote: > >> On 04.03.12 14:02, RW wrote: >>> An alternative would be to be more selective. I'm not sure if this is >>> specific to dovecot but when I copy/move a file in IMAP the new >>> maildir file has

Re: Allowing IMAP users to train spam/ham

2012-03-05 Thread RW
On Mon, 5 Mar 2012 10:54:22 +0100 Matus UHLAR - fantomas wrote: > On 04.03.12 14:02, RW wrote: > >An alternative would be to be more selective. I'm not sure if this is > >specific to dovecot but when I copy/move a file in IMAP the new > >maildir file has the same mtime, but a new epoch time in the

Re: Allowing IMAP users to train spam/ham

2012-03-05 Thread Matus UHLAR - fantomas
LuKreme wrote: I sued to have a setup where IMAP users could put mail into either SPAM or Junk mailboxes to have it auto trained and then I had a script that stepped through and did the training, and it also processed non-new mail in the inbox as ham. On 04.03.12 07:55, xTrade Assessory wrote

Re: Allowing IMAP users to train spam/ham

2012-03-05 Thread Matus UHLAR - fantomas
On 04.03.12 14:02, RW wrote: An alternative would be to be more selective. I'm not sure if this is specific to dovecot but when I copy/move a file in IMAP the new maildir file has the same mtime, but a new epoch time in the file name. What you might do is generate a list of filenames that contain

Re: Allowing IMAP users to train spam/ham

2012-03-04 Thread jdow
On 2012/03/04 11:57, John Hardin wrote: On Sun, 4 Mar 2012, jdow wrote: On 2012/03/04 10:30, LuKreme wrote: On 04 Mar 2012, at 05:36 , xTrade Assessory wrote: > question is if necessary ... Being able to train mis-tagged spam is necessary, yes. I don’t see anyway to process a message in a mai

Re: Allowing IMAP users to train spam/ham

2012-03-04 Thread Jari Fredriksson
4.3.2012 22:44, LuKreme kirjoitti: > Trouble with simply moving the messages about in the shell between Maildirs > is that the courier files don’t get updated properly. > I move my files all the time, and no problems occurred so far. I use Courier too... -- Things past redress and now with m

Re: Allowing IMAP users to train spam/ham

2012-03-04 Thread LuKreme
On 04 Mar 2012, at 12:57 , John Hardin wrote: > On Sun, 4 Mar 2012, jdow wrote: > >> On 2012/03/04 10:30, LuKreme wrote: >>> On 04 Mar 2012, at 05:36 , xTrade Assessory wrote: >>> > question is if necessary ... >>> >>> Being able to train mis-tagged spam is necessary, yes. I don’t see >>> anyw

Re: Allowing IMAP users to train spam/ham

2012-03-04 Thread Jari Fredriksson
4.3.2012 20:49, jdow kirjoitti: > On 2012/03/04 10:30, LuKreme wrote: >> On 04 Mar 2012, at 05:36 , xTrade Assessory wrote: >>> question is if necessary ... >> >> Being able to train mis-tagged spam is necessary, yes. I don’t see >> anyway to process a message in a maildir and then move that messag

Re: Allowing IMAP users to train spam/ham

2012-03-04 Thread John Hardin
On Sun, 4 Mar 2012, jdow wrote: On 2012/03/04 10:30, LuKreme wrote: On 04 Mar 2012, at 05:36 , xTrade Assessory wrote: > question is if necessary ... Being able to train mis-tagged spam is necessary, yes. I don’t see anyway to process a message in a maildir and then move that message. How

Re: Allowing IMAP users to train spam/ham

2012-03-04 Thread jdow
On 2012/03/04 10:30, LuKreme wrote: On 04 Mar 2012, at 05:36 , xTrade Assessory wrote: question is if necessary ... Being able to train mis-tagged spam is necessary, yes. I don’t see anyway to process a message in a maildir and then move that message. How would you do it? bash script with f

Re: Allowing IMAP users to train spam/ham

2012-03-04 Thread LuKreme
On 04 Mar 2012, at 05:36 , xTrade Assessory wrote: > question is if necessary ... Being able to train mis-tagged spam is necessary, yes. I don’t see anyway to process a message in a maildir and then move that message. How would you do it? -- Lister: What d'ya think of Betty? Cat: Betty Rubble?

Re: Allowing IMAP users to train spam/ham

2012-03-04 Thread RW
On Sun, 04 Mar 2012 09:36:25 -0300 xTrade Assessory wrote: > LuKreme wrote: > > On 04 Mar 2012, at 03:55 , xTrade Assessory wrote: > > > >> what do you think of something less complex? > > Yeah, I went with Junk/NotJunk, anything placed in Junk gets > > trained as spam, anything in NotJunk trained

Re: Allowing IMAP users to train spam/ham

2012-03-04 Thread xTrade Assessory
LuKreme wrote: > On 04 Mar 2012, at 03:55 , xTrade Assessory wrote: > >> what do you think of something less complex? > Yeah, I went with Junk/NotJunk, anything placed in Junk gets trained as spam, > anything in NotJunk trained as ham. What I’d like to do though is move the > messages that are in

Re: Allowing IMAP users to train spam/ham

2012-03-04 Thread LuKreme
On 04 Mar 2012, at 03:55 , xTrade Assessory wrote: > what do you think of something less complex? Yeah, I went with Junk/NotJunk, anything placed in Junk gets trained as spam, anything in NotJunk trained as ham. What I’d like to do though is move the messages that are in NotJunk to the inbox m

Re: Allowing IMAP users to train spam/ham

2012-03-04 Thread xTrade Assessory
LuKreme wrote: > I sued to have a setup where IMAP users could put mail into either SPAM or > Junk mailboxes to have it auto trained and then I had a script that stepped > through and did the training, and it also processed non-new mail in the inbox > as ham. Hi what do you think of something

Allowing IMAP users to train spam/ham

2012-03-04 Thread LuKreme
I sued to have a setup where IMAP users could put mail into either SPAM or Junk mailboxes to have it auto trained and then I had a script that stepped through and did the training, and it also processed non-new mail in the inbox as ham. USERROOT="$HOME"; MAILP="Maildir"; J_PATH="$USERROOT/${