Re: R: New domains (was: URIWhois plugin)
--On Thursday, September 27, 2007 7:05 PM +0200 Giampaolo Tomassoni [EMAIL PROTECTED] wrote: The only problem is that a spammer could query it days before it will bulk send, thereby impairing the effectiveness of such approach. I think we need some official data like the domain's creation time: at least, spammers will be mandated to buy domains a couple of month before using them... I recall reading about a scam used by registrars to buy up large blocks of names, then cancel the purchase and incur no penalty, then buy them up again, over and over, to hold them off the market at no cost. How would you detect such a thing?
Re: R: New domains (was: URIWhois plugin)
Quoting Kenneth Porter [EMAIL PROTECTED]: --On Thursday, September 27, 2007 7:05 PM +0200 Giampaolo Tomassoni [EMAIL PROTECTED] wrote: The only problem is that a spammer could query it days before it will bulk send, thereby impairing the effectiveness of such approach. I think we need some official data like the domain's creation time: at least, spammers will be mandated to buy domains a couple of month before using them... I recall reading about a scam used by registrars to buy up large blocks of names, then cancel the purchase and incur no penalty, then buy them up again, over and over, to hold them off the market at no cost. How would you detect such a thing? It's called domain tasting or domain kiting: http://en.wikipedia.org/wiki/Domain_tasting http://www.bobparsons.com/DomainKiting.html Jeff C.
New domains (was: URIWhois plugin)
(The idea below is not mine, someone else (I'm sorry, but I forgot who) wrote about it here (I think) before.) Giampaolo Tomassoni wrote: brand-new domains, Something that could work for this without the problems inherent in using whois or registry databases is to simply check how long ago a domain was first seen beeing used for sending mail or in URIs in mail. (People might allready be doing this locally, but doing it centralized could work better.) A specialized DNS server could be done for this. It'd work something like this: 1: It receives a query. 2: It checks in it's database. 3.a, found in database: * Return result indicating how long ago domain was added. 3.b: not found: * Adds the domain to the database. * Return result indicating new domain. (It might be a good idea to also save last queried time for each domain (meaning 2.a will need to update the database) in order to be able to clean out domains that hasn't been seen for a long time.) In order to be effective, such a DNS list must be used by a lot of different systems spread all over the world and used by different type of organizations. It will also take time time until it can be used in an effective manner, so enough people would have to be using it for some time with very low scores just to seed it. I could probably throw together a proof-of-concept DNS thingy in perl for this, but I don't have the hardware to host it for production use, nor the time to do it properly (perl would probably not be the best language to do it in). The best way might be to actually implement this in an existing DNS-list server, so it could be seeded thorugh queries fopr that list. If, just as an example, SURBL did this, the list would be seeded by all systems allready using SURBL lists, and the results could be included in multi.surbl.org. (Please not, I have no idea if implementing this in SURBLs DNS system is feasible in any way (wr to software, hardware, lunch breaks, or whatever), it was just an example.) Regards /Jonas -- Jonas Eckerman, FSDB Fruktträdet http://whatever.frukt.org/ http://www.fsdb.org/ http://www.frukt.org/
R: New domains (was: URIWhois plugin)
-Messaggio originale- Da: Jonas Eckerman [mailto:[EMAIL PROTECTED] Inviato: giovedì 27 settembre 2007 18.17 A: users@spamassassin.apache.org Oggetto: New domains (was: URIWhois plugin) (The idea below is not mine, someone else (I'm sorry, but I forgot who) wrote about it here (I think) before.) Giampaolo Tomassoni wrote: brand-new domains, Something that could work for this without the problems inherent in using whois or registry databases is to simply check how long ago a domain was first seen beeing used for sending mail or in URIs in mail. (People might allready be doing this locally, but doing it centralized could work better.) A specialized DNS server could be done for this. It'd work something like this: 1: It receives a query. 2: It checks in it's database. 3.a, found in database: * Return result indicating how long ago domain was added. 3.b: not found: * Adds the domain to the database. * Return result indicating new domain. This is really a good idea. The only problem is that a spammer could query it days before it will bulk send, thereby impairing the effectiveness of such approach. I think we need some official data like the domain's creation time: at least, spammers will be mandated to buy domains a couple of month before using them... Giampaolo (It might be a good idea to also save last queried time for each domain (meaning 2.a will need to update the database) in order to be able to clean out domains that hasn't been seen for a long time.) In order to be effective, such a DNS list must be used by a lot of different systems spread all over the world and used by different type of organizations. It will also take time time until it can be used in an effective manner, so enough people would have to be using it for some time with very low scores just to seed it. I could probably throw together a proof-of-concept DNS thingy in perl for this, but I don't have the hardware to host it for production use, nor the time to do it properly (perl would probably not be the best language to do it in). The best way might be to actually implement this in an existing DNS-list server, so it could be seeded thorugh queries fopr that list. If, just as an example, SURBL did this, the list would be seeded by all systems allready using SURBL lists, and the results could be included in multi.surbl.org. (Please not, I have no idea if implementing this in SURBLs DNS system is feasible in any way (wr to software, hardware, lunch breaks, or whatever), it was just an example.) Regards /Jonas -- Jonas Eckerman, FSDB Fruktträdet http://whatever.frukt.org/ http://www.fsdb.org/ http://www.frukt.org/
Re: New domains (was: URIWhois plugin)
Quoting Jonas Eckerman [EMAIL PROTECTED]: (The idea below is not mine, someone else (I'm sorry, but I forgot who) wrote about it here (I think) before.) Giampaolo Tomassoni wrote: brand-new domains, Something that could work for this without the problems inherent in using whois or registry databases is to simply check how long ago a domain was first seen beeing used for sending mail or in URIs in mail. (People might allready be doing this locally, but doing it centralized could work better.) A specialized DNS server could be done for this. It'd work something like this: 1: It receives a query. 2: It checks in it's database. 3.a, found in database: * Return result indicating how long ago domain was added. 3.b: not found: * Adds the domain to the database. * Return result indicating new domain. This is a very good idea, and could be used as a partial substitute for the Day Old Bread list. I particularly like that it could be relatively lightweight and automatic. Naturally there are some complications: 1. What happens if the domain is re-registered before it's delisted? 2. What happens with tasting (kited) domains that get used for 5 days, then unregistered, re-registered, etc. 3. It wouldn't distinguish good domains from bad (but nor does DOB). As Giamoaolo points out, it could be fairly trivially poisoned by the bad guys submitting misleading queries. Remember that many spam URI domains appear before they exist in the whois data or are even delegated from the TLD zones. (The former is a problem for URIWhois too, it would seem.) The quick answer to these issues is that whois is only partially useful. 1. The contact information on spam domains is often false, misleading or stolen as part of identity theft, so it's not always useful. (That said, there are sometimes useful patterns in the contact info.) 2. As mentioned above the whois data is sometimes populated *after* the domains start appearing in spams. Remember that the whois data is still mostly batch processed once or twice a day. Many of the TLD zone files (where the DNS delegations actually come from) are updated in near real time. IOW the whois data can significantly lag usable domains. 3. Nameserver information in whois can be misleading: A. The nameservers and/or their IPs are sometimes changed after the domain is initially registered, either before or after the domain appears in messages. B. The nameservers in the whois are not always the ones that finally resolve a domain. There can be long chains of delegations before getting a final answer. Sometimes only 1 of the whois-listed nameservers actually works. Sometimes none of them work or exist only in DNS caches. C. Spam nameservers have been known to give misleading responses or block access to anti-spammers, malware researchers, etc. There are many other factors besides domain age that can be used to help identify spam domains, and SURBL does use many of them, though we expect to use them more effectively going forward. This is largely without reference to the whois data which for some of the reasons mentioned above are not always reliable or useful. Domain age is only one factor, and considering other factors can make for a significantly more useful blacklist. Cheers, Jeff C.
Re: New domains (was: URIWhois plugin)
2. As mentioned above the whois data is sometimes populated *after* the domains start appearing in spams. Remember that the whois data is still mostly batch processed once or twice a day. Many of the TLD zone files (where the DNS delegations actually come from) are updated in near real time. IOW the whois data can significantly lag usable domains. I'd be tempted to try using the whois data as the basis for a RHSBL not-blacklist (i.e: if the domain name DOESN'T match the RHSBL, then it's not old enough to accept mail from). However, as a practical matter we'd be talking about a multi-gigabyte zone file and I expect that would prove a little problematic for whatever sucker^H^H^H kind soul ended up hosting the thing. -- Dave Pooser Cat-Herder-in-Chief, Pooserville.com ...Life is not a journey to the grave with the intention of arriving safely in one pretty and well-preserved piece, but to slide across the finish line broadside, thoroughly used up, worn out, leaking oil, and shouting GERONIMO!!! -- Bill McKenna
Re: URIWhois plugin
Quoting Giampaolo Tomassoni [EMAIL PROTECTED]: Dears, well, I just did version 0.01 of the URIWhois plugin. Its purpose is mainly to detect some spam containing URIs to sites in brand-new domains, or having some conflict in whois and dns records, or being driven by specific dns servers. So, it is meant to do something I believe someone else is already doing in their SA, but this plugin is completely asynchronous in order to minimize any performance impact. Also, it caches whois results. But the best thing is that, if you run more SA copies on the same computer (in example, you use amavis), when one is asked to issue a whois query for a domain which another copy is already quering, the first SA copy waits for the results obtained by the latter! Finally, it is easily configurable to adapt to your own mileage: you may even avoid whois queries by not using some of the rules. More details by perldoc. Please note this is not stable stuff. It is... well, what's before alpha? The URIWhois plugin needs SA v.3.002003 (or above?) and would surely appreciate a quite recent copy of BerkeleyDB (I'm using 0.31 with v.4.5 of the berkeleydb libraries). You can download it from here: http://www.tomassoni.biz/download/URIWhois-0.01.tar.bz2 (come on, it is 17 KB...). Untar it on the /etc/spamassassin directory and you are (almost) done. Review settings from the /etc/spamassassin/URIWhois.cf file. I would like to have this code reviewed by you, since I'm not that much used to the async thingeries in SA. Enjoy! Giampaolo In principle, this is a good concept; using domain whois data to spot bad domains can be useful. In practice, it's a really, really, really bad idea since the public whois infrastructure is not designed for this kind of high volume use. If many people did it, it would result in an effective DDOS against whois service, even with caching and delays. Please don't do it. It's much better to let URI blacklist operators such as SURBL handle these domains in a centralized way and publish the domain data via our four dozen DNS servers, etc. Jeff C.
Re: URIWhois plugin
Jeff Chan wrote: In principle, this is a good concept; using domain whois data to spot bad domains can be useful. In practice, it's a really, really, really bad idea since the public whois infrastructure is not designed for this kind of high volume use. If many people did it, it would result in an effective DDOS against whois service, even with caching and delays. Please don't do it. It's much better to let URI blacklist operators such as SURBL handle these domains in a centralized way and publish the domain data via our four dozen DNS servers, etc. Jeff C. The other thing is that a LOT of registrars and registries rate limit whois lookups, so it won't work after you've done X lookups in a 24 hour period -- Mr Michele Neylon Blacknight Solutions Hosting Colocation, Brand Protection http://www.blacknight.ie/ http://blog.blacknight.ie/ Tel. 1850 929 929 Intl. +353 (0) 59 9183072 Direct Dial: +353 (0)59 9183090 Fax. +353 (0) 1 4811 763 --- Blacknight Internet Solutions Ltd, Unit 12A,Barrowside Business Park,Sleaty Road,Graiguecullen,Carlow,Ireland Company No.: 370845
R: URIWhois plugin
-Messaggio originale- Da: Jeff Chan [mailto:[EMAIL PROTECTED] In principle, this is a good concept; using domain whois data to spot bad domains can be useful. In practice, it's a really, really, really bad idea since the public whois infrastructure is not designed for this kind of high volume use. If many people did it, it would result in an effective DDOS against whois service, even with caching and delays. Please don't do it. It's much better to let URI blacklist operators such as SURBL handle these domains in a centralized way and publish the domain data via our four dozen DNS servers, etc. How do they handle these domains in a centralized way? Do they simply relay a whois request for not-yet-seen domains? Because in this case they have to tune their whois parsers a bit: dob.sibl.support-intelligence.net, in example, reports both libero.it and tomassoni.biz as being Day One Bread, while it is years they're around... Giampaolo Jeff C.
Re: R: URIWhois plugin
Quoting Giampaolo Tomassoni [EMAIL PROTECTED]: How do they handle these domains in a centralized way? Do they simply relay a whois request for not-yet-seen domains? Because in this case they have to tune their whois parsers a bit: dob.sibl.support-intelligence.net, in example, reports both libero.it and tomassoni.biz as being Day One Bread, while it is years they're around... Day Old Bread has had some errors before. All of .org was blacklisted for a while for example. Aside from the occasional errors, it's a useful list in concept since it shows recently registered domains. However I was referring to URI blacklists such as SURBL.org, not DOB. SURBL doesn't catch everything, but it catches much, and we seek to catch more. It's much better if we do the whois and other queries in a centralized way, do a lot of (quick) testing, then distribute the resulting data as a blacklist which everyone can use as relatively efficient DNS queries or rsync files. Jeff C.
URIWhois plugin
Dears, well, I just did version 0.01 of the URIWhois plugin. Its purpose is mainly to detect some spam containing URIs to sites in brand-new domains, or having some conflict in whois and dns records, or being driven by specific dns servers. So, it is meant to do something I believe someone else is already doing in their SA, but this plugin is completely asynchronous in order to minimize any performance impact. Also, it caches whois results. But the best thing is that, if you run more SA copies on the same computer (in example, you use amavis), when one is asked to issue a whois query for a domain which another copy is already quering, the first SA copy waits for the results obtained by the latter! Finally, it is easily configurable to adapt to your own mileage: you may even avoid whois queries by not using some of the rules. More details by perldoc. Please note this is not stable stuff. It is... well, what's before alpha? The URIWhois plugin needs SA v.3.002003 (or above?) and would surely appreciate a quite recent copy of BerkeleyDB (I'm using 0.31 with v.4.5 of the berkeleydb libraries). You can download it from here: http://www.tomassoni.biz/download/URIWhois-0.01.tar.bz2 (come on, it is 17 KB...). Untar it on the /etc/spamassassin directory and you are (almost) done. Review settings from the /etc/spamassassin/URIWhois.cf file. I would like to have this code reviewed by you, since I'm not that much used to the async thingeries in SA. Enjoy! Giampaolo
