Chip M. wrote: > This snowshoe stuff has been a PITA for a while. > <SNIP> > *** Rob McEwen: *** > Would you be willing to provide your /24 list, for even a short period, > in some sort of plain text format (maybe one CIDR per line?), so those > of us with good hand-classified corpi could try out your data? > > Most of my users are in a shared hosting environment, so they can't use > your list suite as-is. Based on what reliable people have posted, some > of my users should probably benefit from your /24 list. I'd be very > glad to provide you with a list of any FPs I find. :)
When Chip posted this several weeks ago, I had to answer.. "hold on... I'm in the middle of making large-scale improvements the invaluement lists". But improvements are now completed. When I can finally catch my breath, I'll have to discuss the specifics of Chip's reuqest further. In the meantime, I'd like to formally announce the these improvements. Basically, very substantial improvements to the invaluement.com DNSBLs have recently been implemented. Plus, the sign-up page on the web site is now easier to understand, and the 'blog' section of the web site has finally started, with an initial post about these improvements to the lists detailed here: http://dnsbl.invaluement.com/spam-blocker/ <marketing-101>If anyone was thinking about testing out the invaluement lists, but kept putting it off... now is a great time to start because, along with these recent improvements, there is also now a special deal where *anyone who converts their free test into a subscription during the month of March/09... will get a free month* added to their first billing period. And *if such a subscriber chooses yearly billing, 4 free months will be added to to their first billing period*. Unfortunately, time is short and in order to do this, the free portion of the testing period would be shorted a bit because the subscription would have to start in March/09. So if this is something that interests you, I recommend getting started as soon as possible so you'll get as many free testing days as possible before needing to make a decision which would take advantage of those free months. The longer you wait, the shorter those free testing days would be if you wanted to get those free months!</marketing-101> To save you a click, and keep this info on-list, here is that blog post: ************************************** *DETAILS ABOUT RECENT INVALUEMENT.COM IMPROVEMENTS* The most substantial improvements were made to ivmSIP/24. So I'll cover that last. First, I'll briefly address improvements to ivmSIP and ivmURI. *ivmSIP IMPROVEMENTS:* During recent work on ivmSIP/24, a “bug” was discovered in the ivmSIP portion of the engine which was causing needless False Negatives. This had been there since the /beginning/ of the invaluement lists! Fixing this should lead to additional ivmSIP effectiveness. (I about fell out of my seat when I spotted the bug!) So whatever you've thought/read/experienced about ivmSIP... it is now even better! *ivmURI IMPROVEMENTS:* Large improvements to the ivmURI engine were made. For example, most good DNSBLs will employ some type of False Positives-prevention filters which then lead to unintended False Negatives (items that really should have been blacklisted). ivmURI is no different and those unintended FNs produced by False Positives-prevention filters are a fact of life. But I'm happy to report that the ivmSIP FP-prevention filters have been greatly improved so as to continue preventing FPs just as well as ever, but those unintended FNs have now been greatly trimmed. This should boost ivmURI’s “catch rates” noticeably, if not substantially. *ivmSIP/24 IMPROVEMENTS:* This is where the vast majority of improvements occurred. ivmSIP/24 is almost like a whole new list. The invaluement 'engine' now has an automated system which enumerates through all 256 possible IPs for each /24 block and individually evaluates each IP for possible /exclusion/ from ivmSIP/24. The improvements made are substantial because ivmSIP/24 now does NOT have listed many subranges allocated to innocent senders in those /24 blocks which were split between innocent senders and spammers. Yet, at the same time, such subranges allocated to spammers remain listed on ivmSIP/24, with many IPs /preemptively/ listed. Additionally, we now have the ability to manually 'whitelist' individual IPs from ivmSIP/24 without having to delist the whole block. Even better, when that happens, the IP is NOT in our regular whitelist and, therefore, has just as good a chance to get listed in the regular ivmSIP list as any other IP. So this should probably be called an ivmSIP/24 “exemption list”, not a whitelist. Should that IP ever get caught sending spam, then all bets are off and its existance on the ivmSIP/24 “exemption list” is then completely ignored. This helps prevent spammers from lying about IPs and adding IPs to the “exemption list” where they really just haven't yet used that IP for spamming yet. In the past, there have been many gut-wrenching situations where an innocent sender is using an IP located on a /24 block where the vast majority of IPs on that block are used by egregious spammers. In many cases, we’ve had to ignore the removal request for those 'innocent' senders. In other cases, we’ve had to remove the whole /24 block from ivmSIP/24 on account of the innocent senders on that block, even where most of the block was used for spamming. Along the way, a common theme is that the innocent sender will complain (in their ivmSIP/24 removal request) that their outbound e-mail is getting blocked by [dedacted ISP-01], [dedacted ISP-02], [dedacted ISP-03], [dedacted ISP-04], [dedacted ISP-05], [dedacted ISP-06], [dedacted ISP-07], [dedacted ISP-08]. (Typically, prior to these new improvements to ivmSIP/24, the 'innocent' sender thinks that ivmSIP/24 is at fault for deliverability problems to these ISPs, because ivmSIP/24 was often the /only/ list to show up on public lookup forms as having this IP blacklisted.) Some such complaints are probably “chaff” and might not be indicators of /24 blocking by some of these ISPs. To be extra clear, I've very unsure if all of these ISPs I mentioned do this. These are just names that have come up from what were really innocent senders. But [dedacted ISP-01] comes up often and consistently. So I feel very confident that [dedacted ISP-01] aggressively blocks /24 blocks in a manner similar to using (the old!) ivmSIP/24 for outright blocking (but with their own internal /24 blacklist). I'm also fairly sure that [dedacted ISP-02], [dedacted ISP-06], [dedacted ISP-05] does this. Sorry, I couldn't name names in public. But all of these are 'household name' ISPs. The bottom line is that many of these ISPs use /24 blocking similar to the old ivmSIP/24. And the new ivmSIP/24 technology has now leapfrogged their technology. I dare say that ivmSIP/24 is now a whole generation ahead of whatever they are doing internally. The “nuts and bolts” of the matter is that there will be many innocent senders continuing to get blocked by some of those ISP's internal /24 lists... but where /most/ of those same innocent senders are NOT listed on the newly improved ivmSIP/24... but where the actual spammers on each of those /24 blocks are still /preemptively/ blacklisted on ivmSIP/24. Also, some of our subscribers who are not particularly concerned about FPs (and this is a bit atypical for our subscribers) have taken the news about ivmSIP/24 and 'yawned'. But I should remind everyone that many /24 blocks which are crawling with snowshoe spammers, but have been previously removed from ivmSIP/24 to protect innocent senders... but these blocks are now re-listed on the new ivmSIP/24 list (but with the innocent senders’ IPs /not/ listed). Since these are now added back to ivmSIP/24, then that means that the new and improved ivmSIP/24 will reduce both FPs AND FNs!* * *QUESTION:* Does this mean that ivmSIP/24 is now safe to use for outright blocking? I can safely say that the new ivmSIP/24 is now order of magnitudes safer for outright blocking than the old ivmSIP/24. But subscribers, and especially larger ISPs, should still use caution. And, in general, everyone should outright block on any dnsbl “at their own risk”, and only after evaluating the track record for that dnsbl. At the least, in a scoring system, I suggest that current subscribers increase the ivmSIP/24 score by a point or two over what I'd have scored the old version of ivmSIP/24. But I wouldn't recommend scoring it as high or higher than ivmSIP. *And if you haven't ever implemented the **invaluement DNSBL**, now is a great time to start!* -- Rob McEwen http://dnsbl.invaluement.com/ r...@invaluement.com +1 (478) 475-9032
