As Pete already provided input on this. I am not going to prolix the answer other than to say when implementing Message Sniffer we abided by the Pete's advice "Since many legitimate ISPs also produce a lot of spam it might be useful to apply a bias to this weight so that these systems appear closer to zero." So currently we do not allow for a negative value as a BASEPOINT, with that said if you think it is really important to be able to use a negative value as you have described in your post, let me know and I can add it to the dev list.
David Barker VP Operations Declude Your Email security is our business 978.499.2933 office 978.988.1311 fax <mailto:dbar...@declude.com> dbar...@declude.com From: supp...@declude.com [mailto:supp...@declude.com] On Behalf Of Andy Schmidt Sent: Saturday, May 01, 2010 1:51 PM To: declude.junkmail@declude.com Subject: RE: [Declude.JunkMail] Sniffer IP Reputation for "white" listing Hi Pete, Funny - our messages overlapped. But I'm glad I was on the right track with my suspicions. Hopefully this will help Declude to refine things. >> a better way to do it would be to scale the result so that from 0 to -1 the "negative" weight (let's pick a factor of 5) would rise linearly from 0 to -5 and similarly a positive going reputation would scale linearly from 0 to +5 as the API result scaled from 0 to +1. << Right - that's the same scheme I just pointed out to Dave myself - except in my case you could pick a distinct factor for the "-" vs. the "+" side of the scale (because Declude already has that option anyhow) (( Abs(Reputation Value) * 10 ) - Base Value) * [Pos or Neg]WeightFactor = Final Weight For this line in the Declude config: IPREPUTATION SNFIPREP x 0 2 -1 it would results in weights between +20 and -10, e.g.: Reputation 0.0: ( ( 0.0 * 10 ) - 0 ) * 2 = 0 Reputation 0.3: ( ( 0.3 * 10 ) - 0 ) * 2 = 6 Reputation 1.0: ( ( 1.0 * 10 ) - 0 ) * 2 = 20 Reputation -0.3: ( ( 0.3 * 10 ) - 0 ) * -1 = -3 Reputation -1.0: ( ( 1.0 * 10 ) - 0 ) * -1 = -10 Here's an important question, though: Do you have a distribution chart for the reputation scale? It of course makes a HUGE different, whether the distribution of reputations reported for the inflow of email is evenly distributed between -1.0 and 0.1, or whether it is a bell curve where 80% are in the "center" area, or whether it's some sort of exponential curve that has very few with "good" reputation, a modest amount around the 0 point, and then expentionally increasing towards the bad and turn reputations? This way one could decide what factors to use for the + and - sides and where to set the "mid" point (Declude allows you to shift the mid-point left and right. >> I'm guessing on how that test is implemented, but if I've guessed correctly then -0.8 would certainly be a good WHITE set point.<< Thank you - that means in their "default" (sample) config file, they really should adjust the midpoint away from "0" to "-8" (they multiply the reputation scale by 10 to be able to work with integers) IPREPUTATION SNFIPREP x 0 2 -1 probably to IPREPUTATION SNFIPREP x -8 2 -1 but I'd have to check with Dave to see if "-8" will indeed set the midpoint to -0.8 or if the sign has to be reversed. Thanks for taking the time to help all of us understand Sniffer in the context of the Declude integration. I'm very happy that Declude took the time and integrated the product. I just would like to make sure it comes with an implementation sample that is a good enough compromise for "day-to-day" use. Best Regards, Andy -----Original Message----- From: supp...@declude.com [mailto:supp...@declude.com] On Behalf Of Pete McNeil Sent: Saturday, May 01, 2010 11:57 AM To: declude.junkmail@declude.com Subject: Re: [Declude.JunkMail] Sniffer IP Reputation for "white" listing On 4/30/2010 9:32 PM, Andy Schmidt wrote: <snip/> > But your documentation of the reputation system has a graph that shows that > there is yet another category: "WHITE". > I don't know the details of Declude's impelementation. Presumably they could (or maybe even do) implement WHITE. > The SNFIPREP tests does offer the ability to define at what decimal value > (between -1 and +1, in .1 increments) a weight can be subtracted. But the > question is - is that SENSIBLE use of your reputation database? Per example, > could -0.8 be a sensible threshold to give an email "credit" for coming from > a reputable IP source? > I'm guessing on how that test is implemented, but if I've guessed correctly then -0.8 would certainly be a good WHITE set point. My guess is based on using a combined score value from the IP reputation that combines the confidence figure and the probability figure. In that case only a strongly negative p coupled with a strong c would result in a -0.8. > Or is it better to let the "good" reputation be considered AFTER the content > scan and then use the "combined" exit code? > As I understand it Declude uses a wheighting system --- except for some short-circuit abilities that means all tests are run, their scores are added together, and then the total is used to determine the disposition of the message. I don't think there is an 'AFTER' in this case. The IP reputation test is useful in cases where a message might be too new to hit a pattern match and where the IP reputation is not quite strong enough to be in one of the GBUdb envelopes. In such a case it might be useful to combine the 'analog' reputation score with the scores from other tests to push the message over the fence one way or another... at least that's how the test was designed to work in the API we provide. It sounds like you're describing the IP Reputation test as having thresholds. That's an interesting way to do it (I haven't looked to see if it is actually that way)... a better way to do it would be to scale the result so that from 0 to -1 the "negative" weight (let's pick a factor of 5) would rise linearly from 0 to -5 and similarly a positive going reputation would scale linearly from 0 to +5 as the API result scaled from 0 to +1. The API result holds 0 as meaning "I don't know" --- either because the confidence figure (c) is 0 or because the probability figure (p) is 0 (meaning a 50% chance of spam or ham). The farther away from 0 you get the more certain the statistics. Hope this helps, _M --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to imail...@declude.com, and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to imail...@declude.com, and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to imail...@declude.com, and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com.
