Re: Domain ages (was Re: SPAM from a registrar)
On Mon, 9 Jun 2014, Rob McEwen wrote: Domain age is a good metric to factor in. But I'm always fascinated with some people's desire to block all messages with extremely new domains. Keep in mind that many large and famous businesses... who have fairly good mail sending practices... sometimes launch a new products complete with links to very newly registered domains. Same is often true for ... Or for public research organizations which are often reformed by the Government, with change of name and consequential change of domain (even if the IP of the DNS and MX is unchanged :-)) Take my case, I've been working at the same physical place since 1982 and the name of my institute or of the organization it belongs to has changed about 7 times. And it does not only occur in this country (Italy), I've seen (mainly dealing with mailing list re-subscriptions) similar changes at least in France and UK .. -- Lucio Chiappetti - INAF/IASF - via Bassini 15 - I-20133 Milano (Italy) For more info : http://www.iasf-milano.inaf.it/~lucio/personal.html
Re: Domain ages (was Re: SPAM from a registrar)
Quoting Lucio Chiappetti lu...@lambrate.inaf.it: On Mon, 9 Jun 2014, Rob McEwen wrote: Domain age is a good metric to factor in. But I'm always fascinated with some people's desire to block all messages with extremely new domains. Keep in mind that many large and famous businesses... who have fairly good mail sending practices... sometimes launch a new products complete with links to very newly registered domains. Same is often true for ... Or for public research organizations which are often reformed by the Government, with change of name and consequential change of domain (even if the IP of the DNS and MX is unchanged :-)) Take my case, I've been working at the same physical place since 1982 and the name of my institute or of the organization it belongs to has changed about 7 times. And it does not only occur in this country (Italy), I've seen (mainly dealing with mailing list re-subscriptions) similar changes at least in France and UK .. Not saying this doesn't happen. But also, how often does someone register a domain, move all their users to the new domain, have the server all reconfigured to use this new domain, all within the first day? I know personally, I have always taken at least a week to do it, mainly just to make sure I didn't miss anything, and to double check everything as I go. The Last thing I do is force users to change their email addresses.
Re: Domain ages (was Re: SPAM from a registrar)
On 06/10/2014 12:28 PM, Patrick Domack wrote: Not saying this doesn't happen. But also, how often does someone register a domain, move all their users to the new domain, have the server all reconfigured to use this new domain, all within the first day? I know personally, I have always taken at least a week to do it, mainly just to make sure I didn't miss anything, and to double check everything as I go. The Last thing I do is force users to change their email addresses. domains don't have to have users on them. coming up film sites, parents setting up sweet16 sites, wedding sites, cosmetic vendors, art festivals, etc, etc, TONS of etc, use new domains for marketing, often seen in mail even before DNS is has fully replicated.
Re: Domain ages (was Re: SPAM from a registrar)
Quoting Axb axb.li...@gmail.com: On 06/10/2014 12:28 PM, Patrick Domack wrote: Not saying this doesn't happen. But also, how often does someone register a domain, move all their users to the new domain, have the server all reconfigured to use this new domain, all within the first day? I know personally, I have always taken at least a week to do it, mainly just to make sure I didn't miss anything, and to double check everything as I go. The Last thing I do is force users to change their email addresses. domains don't have to have users on them. coming up film sites, parents setting up sweet16 sites, wedding sites, cosmetic vendors, art festivals, etc, etc, TONS of etc, use new domains for marketing, often seen in mail even before DNS is has fully replicated. Yes, anything is possible. I have yet, to see any ligit email though, I'm sure I will a few times a year. I have seen email before dns/whois even is updated. But personally, one should work to establish their reputation before blasting out emails. You have to do this when moving ip addresses, and also for domains, though not as many servers track domain reputation as much as ip reputation.
Re: Domain ages (was Re: SPAM from a registrar)
On 06/10/2014 04:14 PM, Patrick Domack wrote: Quoting Axb axb.li...@gmail.com: On 06/10/2014 12:28 PM, Patrick Domack wrote: Not saying this doesn't happen. But also, how often does someone register a domain, move all their users to the new domain, have the server all reconfigured to use this new domain, all within the first day? I know personally, I have always taken at least a week to do it, mainly just to make sure I didn't miss anything, and to double check everything as I go. The Last thing I do is force users to change their email addresses. domains don't have to have users on them. coming up film sites, parents setting up sweet16 sites, wedding sites, cosmetic vendors, art festivals, etc, etc, TONS of etc, use new domains for marketing, often seen in mail even before DNS is has fully replicated. Yes, anything is possible. I have yet, to see any ligit email though, I'm sure I will a few times a year. I have seen email before dns/whois even is updated. But personally, one should work to establish their reputation before blasting out emails. You have to do this when moving ip addresses, and also for domains, though not as many servers track domain reputation as much as ip reputation. you honestly expect marketing drones to understand/care? All URI BLs I know of (SURBL/URIBL/DBL/Invaluement/etc) check track domain reputation otherwise they'd be unusable. Their listings are not blind - they all have their secret sauce to process before listing a domain.
Re: Domain ages (was Re: SPAM from a registrar)
Quoting Axb axb.li...@gmail.com: On 06/10/2014 04:14 PM, Patrick Domack wrote: Quoting Axb axb.li...@gmail.com: On 06/10/2014 12:28 PM, Patrick Domack wrote: Not saying this doesn't happen. But also, how often does someone register a domain, move all their users to the new domain, have the server all reconfigured to use this new domain, all within the first day? I know personally, I have always taken at least a week to do it, mainly just to make sure I didn't miss anything, and to double check everything as I go. The Last thing I do is force users to change their email addresses. domains don't have to have users on them. coming up film sites, parents setting up sweet16 sites, wedding sites, cosmetic vendors, art festivals, etc, etc, TONS of etc, use new domains for marketing, often seen in mail even before DNS is has fully replicated. Yes, anything is possible. I have yet, to see any ligit email though, I'm sure I will a few times a year. I have seen email before dns/whois even is updated. But personally, one should work to establish their reputation before blasting out emails. You have to do this when moving ip addresses, and also for domains, though not as many servers track domain reputation as much as ip reputation. you honestly expect marketing drones to understand/care? All URI BLs I know of (SURBL/URIBL/DBL/Invaluement/etc) check track domain reputation otherwise they'd be unusable. Their listings are not blind - they all have their secret sauce to process before listing a domain. So, we are unwilling to look into any new ideas cause there might be an issue? that we haven't scoped or checked into? How is progress made, when your unwilling to check and collect stats and figures. This was meant to be another metric that could, or might not be used. I personally got tired of everyone talking about it, many shooting it down, and NO ONE actually looking into it, and reporting real stats about it. Personally, I thought it was a pointless test, but it is proving useful. Does it single handily solve spam and has no side effects? No, but if you find that solution, you will be rich.
Re: Domain ages (was Re: SPAM from a registrar)
On 6/10/2014 10:21 AM, Axb wrote: All URI BLs I know of (SURBL/URIBL/DBL/Invaluement/etc) check track domain reputation otherwise they'd be unusable. Their listings are not blind - they all have their secret sauce to process before listing a domain. Absolutely. As Axb and KAM and others stated, a very young domain age is too dangerous to outright block or score high on... but might be a good factor or good for combining with other rules. Also, if anyone does see spam that contain domains in the clickable links where that spam should have been blocked, but was not... then check the domain contained within the spam again the lookup found at http://multirbl.valli.org and/or http://mxtoolbox.com/blacklists.aspx (some months ago, MX Toolbox upgraded their system to check domains against URI/domain blacklists. In some cases, this could have been a game of inches where your user caught the tip of the spear and received the very first spams in a spam campaign that otherwise was quickly listed by the well known URI BLs. However, you may find that one or two good URI BLs are simply not implemented in your system--where that would have made all the difference! Those lookup forms will point you in the right direction. The same can also be true for checking sending IPs--then reviewing your current mix of sender's IP dnsbls (aka RBLs). Of course, don't fall into the trap of using a BL that catches much, but has too many FPs. But the list of URI BLs that Axb gave above are all extremely low-FP URI blacklists. -- Rob McEwen +1 (478) 475-9032
Re: Domain ages (was Re: SPAM from a registrar)
On 6/10/2014 10:34 AM, Patrick Domack wrote: So, we are unwilling to look into any new ideas cause there might be an issue? that we haven't scoped or checked into? Patrick, I don't think Axe was arguing against this idea.. I think he was arguing against irrational exuberance by some who may be taking this idea to the point of outright blocking (or high scoring) on it, which would generate significant FPs. His examples were solid real world examples that DO happen and WOULD FP if this idea were taken too far. But using extremely-young-domain-age for low scoring or in combination with other rules could be very helpful. -- Rob McEwen +1 (478) 475-9032
Re: Domain ages (was Re: SPAM from a registrar)
Quoting Rob McEwen r...@invaluement.com: On 6/10/2014 10:21 AM, Axb wrote: All URI BLs I know of (SURBL/URIBL/DBL/Invaluement/etc) check track domain reputation otherwise they'd be unusable. Their listings are not blind - they all have their secret sauce to process before listing a domain. Absolutely. As Axb and KAM and others stated, a very young domain age is too dangerous to outright block or score high on... but might be a good factor or good for combining with other rules. Also, if anyone does see spam that contain domains in the clickable links where that spam should have been blocked, but was not... then check the domain contained within the spam again the lookup found at http://multirbl.valli.org and/or http://mxtoolbox.com/blacklists.aspx (some months ago, MX Toolbox upgraded their system to check domains against URI/domain blacklists. In some cases, this could have been a game of inches where your user caught the tip of the spear and received the very first spams in a spam campaign that otherwise was quickly listed by the well known URI BLs. However, you may find that one or two good URI BLs are simply not implemented in your system--where that would have made all the difference! Those lookup forms will point you in the right direction. The same can also be true for checking sending IPs--then reviewing your current mix of sender's IP dnsbls (aka RBLs). Of course, don't fall into the trap of using a BL that catches much, but has too many FPs. But the list of URI BLs that Axb gave above are all extremely low-FP URI blacklists. In my case, Yes, I am using all the above and more. I had a user that normally never gets spam, started receiving around 20 per day, that where not marked. I found that around 18per day of these where from a new domain. These did appear on multirbl.valli.org lists, like invaluement, and uribl after a day or two. I hadn't seen them hit dbl or surbl though. This is what caused me to seriously look into if this method was useful, just greylisting the email for a day, would cause a huge benifit, for new domains, without causing an extreem backlash. There are all kinds of way to use the infomation. I just don't understand why people are so against it, cause it's not 100% foolproof. Nothing about marking spam is 100% foolproof.
Re: Domain ages (was Re: SPAM from a registrar)
On 06/10/2014 04:34 PM, Patrick Domack wrote: Quoting Axb axb.li...@gmail.com: On 06/10/2014 04:14 PM, Patrick Domack wrote: Quoting Axb axb.li...@gmail.com: On 06/10/2014 12:28 PM, Patrick Domack wrote: Not saying this doesn't happen. But also, how often does someone register a domain, move all their users to the new domain, have the server all reconfigured to use this new domain, all within the first day? I know personally, I have always taken at least a week to do it, mainly just to make sure I didn't miss anything, and to double check everything as I go. The Last thing I do is force users to change their email addresses. domains don't have to have users on them. coming up film sites, parents setting up sweet16 sites, wedding sites, cosmetic vendors, art festivals, etc, etc, TONS of etc, use new domains for marketing, often seen in mail even before DNS is has fully replicated. Yes, anything is possible. I have yet, to see any ligit email though, I'm sure I will a few times a year. I have seen email before dns/whois even is updated. But personally, one should work to establish their reputation before blasting out emails. You have to do this when moving ip addresses, and also for domains, though not as many servers track domain reputation as much as ip reputation. you honestly expect marketing drones to understand/care? All URI BLs I know of (SURBL/URIBL/DBL/Invaluement/etc) check track domain reputation otherwise they'd be unusable. Their listings are not blind - they all have their secret sauce to process before listing a domain. So, we are unwilling to look into any new ideas cause there might be an issue? that we haven't scoped or checked into? How is progress made, when your unwilling to check and collect stats and figures. This was meant to be another metric that could, or might not be used. I personally got tired of everyone talking about it, many shooting it down, and NO ONE actually looking into it, and reporting real stats about it. Personally, I thought it was a pointless test, but it is proving useful. Does it single handily solve spam and has no side effects? No, but if you find that solution, you will be rich. Your ideas/approach has been dealt with a decade ago - they're not new. While it may work for you, they may not scale on a global user base.
Re: Domain ages (was Re: SPAM from a registrar)
On 06/10/2014 05:11 PM, Patrick Domack wrote: There are all kinds of way to use the infomation. I just don't understand why people are so against it, cause it's not 100% foolproof. Nobody is against the idea, problem is scalability and trust. To make domain age usable, the BLs I mentioned make use of it as well as many other daata points to gain trust that a listing won' tbite the globe, as well as they can. Consider certain factors wich *can* contribute to delay in listings produce a positive hit,for example, mirror lag due to rsync, negative TTL, etc. as reasosn why you seem to see these domains being listed after you got the spams. (If your size/budget permits, datafeeds would probably help a lot) For a small site doing a few whois lookups/hour it may work, but what if suddenly an ISP/ASP doing many thousands of msgs/sec would implement this?
Re: Domain ages (was Re: SPAM from a registrar)
Quoting Axb axb.li...@gmail.com: On 06/10/2014 05:11 PM, Patrick Domack wrote: There are all kinds of way to use the infomation. I just don't understand why people are so against it, cause it's not 100% foolproof. Nobody is against the idea, problem is scalability and trust. To make domain age usable, the BLs I mentioned make use of it as well as many other daata points to gain trust that a listing won' tbite the globe, as well as they can. Consider certain factors wich *can* contribute to delay in listings produce a positive hit,for example, mirror lag due to rsync, negative TTL, etc. as reasosn why you seem to see these domains being listed after you got the spams. (If your size/budget permits, datafeeds would probably help a lot) For a small site doing a few whois lookups/hour it may work, but what if suddenly an ISP/ASP doing many thousands of msgs/sec would implement this? I did consider those factors, and they where not the problem. I do rsync the data feeds locally, and feeds did not contain the lookups till hours later. It wasn't a negative ttl issue, as the ttl is non-existant for these lookups I fail to understand why you would be doing thousands of whois lookups per second. You see that many new domain names per second? Mostly it's the same domain names over and over again, and a few new ones per day. Domains don't expire, moved around, and updated a lot, and even if it did, that isn't really much a concern. To cache this infomation for atleast 3 years, would be fine, likely even longer. Also, the point of having a central body do this, would cause the cached results to be even better, and less lookups needed. I'm not a huge isp, but I don't seem to be any where near as tiny as you suggest.
Re: Domain ages (was Re: SPAM from a registrar)
On 06/10/2014 06:51 PM, Patrick Domack wrote: Quoting Axb axb.li...@gmail.com: On 06/10/2014 05:11 PM, Patrick Domack wrote: There are all kinds of way to use the infomation. I just don't understand why people are so against it, cause it's not 100% foolproof. Nobody is against the idea, problem is scalability and trust. To make domain age usable, the BLs I mentioned make use of it as well as many other daata points to gain trust that a listing won' tbite the globe, as well as they can. Consider certain factors wich *can* contribute to delay in listings produce a positive hit,for example, mirror lag due to rsync, negative TTL, etc. as reasosn why you seem to see these domains being listed after you got the spams. (If your size/budget permits, datafeeds would probably help a lot) For a small site doing a few whois lookups/hour it may work, but what if suddenly an ISP/ASP doing many thousands of msgs/sec would implement this? I did consider those factors, and they where not the problem. I do rsync the data feeds locally, and feeds did not contain the lookups till hours later. It wasn't a negative ttl issue, as the ttl is non-existant for these lookups When you come up with a couple of such cases, please post them here as quickly as you can so BL ops or users lurking here can check their logs and maybe compare results. I fail to understand why you would be doing thousands of whois lookups per second. You see that many new domain names per second? Mostly it's the same domain names over and over again, and a few new ones per day. You do lookups on URIS in your mailflow right? so you do it for HAM/SPAM Domains don't expire, moved around, and updated a lot, and even if it did, that isn't really much a concern. To cache this infomation for atleast 3 years, would be fine, likely even longer. Check keep track of daily changes and you'll be surprised how often stuff gets moved around. Also, the point of having a central body do this, would cause the cached results to be even better, and less lookups needed. if found...ok. if not found negative TTL applies and short TTL means evne more lookups. I'm not a huge isp, but I don't seem to be any where near as tiny as you suggest. I'm not assuming/suggesting anything
Re: Domain ages (was Re: SPAM from a registrar)
On Mon, 9 Jun 2014 22:44:22 +0200
Matthias Leisi matth...@leisi.net wrote:
I still have an experimental DNS server (written in Perl) lying
around that this more-or-less what is described here. The overall
system would need a bit more thought, though.
Attached is a hacky proof-of-concept script that stores state in
Berkeley DB. You query something.com.da.example.com and get back a
TXT record with the UNIX time in seconds or an A record with the UNIX
time encoded in the A record. (This time is the time in seconds since
Jan 1 1970 00:00 UTC when the domain was first queried.)
The script only handles com, net and org top-level domains. It also
only looks at the domain label just before com, net and org so that
foo.com and sub.foo.com are both treated as foo.com
It sets the TTL of returned records to 14 days, so if you put this behind
a caching name server like unbound, it might even work OK under
reasonably heavy load.
Regards,
David.
===
#!/usr/bin/perl
use strict;
use warnings;
use DB_File;
use Net::DNS::Nameserver;
my %hash;
# Replace this with the path of your DB
my $handle = tie %hash, 'DB_File', 'domain-age.db';
# Adjust settings below as needed...
my $ns = new Net::DNS::Nameserver(
LocalAddr = ['127.0.0.1'],
LocalPort = '5354',
ReplyHandler = \handler,
Verbose = 0,
Truncate = 0,
);
$ns-main_loop();
exit(1);
sub chunk_to_addr
{
my ($chunk) = @_;
my $d = $chunk 255; $chunk /= 256;
my $c = $chunk 255; $chunk /= 256;
my $b = $chunk 255; $chunk /= 256;
my $a = $chunk 255;
return $a.$b.$c.$d;
}
sub handler
{
my ($qname, $qclass, $qtype, $peerhost, $query, $conn) = @_;
my (@ans, @auth, @add);
# Adjust qname regex as needed
if ($qname !~ /([^.]+)\.(com|net|org)\.da\.example\.com$/i) {
return ('REFUSED', \@ans, \@auth, \@add, {aa = 1 });
}
if ($qtype ne 'TXT' $qtype ne 'A') {
return ('NXDOMAIN', \@ans, \@auth, \@add, {aa = 1 });
}
my $chunk = lc($1.$2);
if (!exists($hash{$chunk})) {
$hash{$chunk} = time();
# FIXME: Maybe don't sync too often? Keep track and only
# sync every 10 seconds?
$handle-sync();
}
if ($qtype eq 'TXT') {
push(@ans, new Net::DNS::RR(name = $qname,
ttl = 86400 * 14,
type = 'TXT',
txtdata = $hash{$chunk}));
} elsif ($qtype = 'A') {
push(@ans, new Net::DNS::RR(name = $qname,
ttl = 86400 * 14,
type = 'A',
address =
chunk_to_addr($hash{$chunk})));
}
return ('NOERROR', \@ans, \@auth, \@add, {aa = 1 });
}
Re: Domain ages (was Re: SPAM from a registrar)
Quoting Axb axb.li...@gmail.com: On 06/10/2014 06:51 PM, Patrick Domack wrote: Quoting Axb axb.li...@gmail.com: On 06/10/2014 05:11 PM, Patrick Domack wrote: There are all kinds of way to use the infomation. I just don't understand why people are so against it, cause it's not 100% foolproof. Nobody is against the idea, problem is scalability and trust. To make domain age usable, the BLs I mentioned make use of it as well as many other daata points to gain trust that a listing won' tbite the globe, as well as they can. Consider certain factors wich *can* contribute to delay in listings produce a positive hit,for example, mirror lag due to rsync, negative TTL, etc. as reasosn why you seem to see these domains being listed after you got the spams. (If your size/budget permits, datafeeds would probably help a lot) For a small site doing a few whois lookups/hour it may work, but what if suddenly an ISP/ASP doing many thousands of msgs/sec would implement this? I did consider those factors, and they where not the problem. I do rsync the data feeds locally, and feeds did not contain the lookups till hours later. It wasn't a negative ttl issue, as the ttl is non-existant for these lookups When you come up with a couple of such cases, please post them here as quickly as you can so BL ops or users lurking here can check their logs and maybe compare results. I fail to understand why you would be doing thousands of whois lookups per second. You see that many new domain names per second? Mostly it's the same domain names over and over again, and a few new ones per day. You do lookups on URIS in your mailflow right? so you do it for HAM/SPAM Domains don't expire, moved around, and updated a lot, and even if it did, that isn't really much a concern. To cache this infomation for atleast 3 years, would be fine, likely even longer. Check keep track of daily changes and you'll be surprised how often stuff gets moved around. Also, the point of having a central body do this, would cause the cached results to be even better, and less lookups needed. if found...ok. if not found negative TTL applies and short TTL means evne more lookups. I'm not a huge isp, but I don't seem to be any where near as tiny as you suggest. I'm not assuming/suggesting anything I'm not interested in how much stuff gets moved around, if a domain has been registered, and been moved around, it will have a reputation. So I don't really care if the data is 100% accurate or up to date. I'm not sure why negative ttl would cause more whois lookups? yes it will cause more dns lookups, but those are not an issue, expecially if you have a local data feed available, if you do, set your negative ttl to 5seconds.
Domain ages (was Re: SPAM from a registrar)
On Mon, 09 Jun 2014 14:24:19 -0400 Patrick Domack patric...@patrickdk.com wrote: That could be easily done. Only issue is, if you trust the distributed lookups to have accurate infomation. I suppose we could build in a trust system, where if enough distributed clients upload the same info, it could be trusted. There's a company that offers a domain-age-like service: https://www.farsightsecurity.com/Services/NOD/ Their approach is interesting (they receive a huge volume of DNS traffic and keep track of domain lookups that are newly seen.) Their price for practical volumes of lookups, unfortunately, is ridiculously expensive, which has prevented us from pursuing this any further. Regards, David.
Re: Domain ages (was Re: SPAM from a registrar)
On 6/9/2014 2:38 PM, David F. Skoll wrote: On Mon, 09 Jun 2014 14:24:19 -0400 Patrick Domack patric...@patrickdk.com wrote: That could be easily done. Only issue is, if you trust the distributed lookups to have accurate infomation. I suppose we could build in a trust system, where if enough distributed clients upload the same info, it could be trusted. There's a company that offers a domain-age-like service: https://www.farsightsecurity.com/Services/NOD/ Their approach is interesting (they receive a huge volume of DNS traffic and keep track of domain lookups that are newly seen.) Their price for practical volumes of lookups, unfortunately, is ridiculously expensive, which has prevented us from pursuing this any further. I think the core issue is that age of domains is a good indicator of spam. So there is merit in building a distributed look-up system using SA. I have more ideas than resources, of course...
Re: Domain ages (was Re: SPAM from a registrar)
On Mon, 9 Jun 2014, Kevin A. McGrail wrote: So there is merit in building a distributed look-up system using SA. Distributed lookup of *what*, though? Can you clarify that part of your idea? Are you referring to distributed whois queries for a domain name, to determine its age? -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- Gun Control laws aren't enacted to control guns, they are enacted to control people: catholics (1500s), japanese peasants (1600s), blacks (1860s), italian immigrants (1911), armenians (1911), the irish (1920s), jews (1930s), blacks (1960s), the poor (always) --- 739 days since the first successful private support mission to ISS (SpaceX)
Re: Domain ages (was Re: SPAM from a registrar)
Domain age is a good metric to factor in. But I'm always fascinated with some people's desire to block all messages with extremely new domains. (NOT saying that this applies to everyone who posted on this thread!) Keep in mind that many large and famous businesses... who have fairly good mail sending practices... sometimes launch a new products complete with links to very newly registered domains. Same is often true for advertisments for things like rock concerts, etc. Or web sites that deal with specific events or hot-topic political issues that appeared out of nowhere. Yes, some of these are UBE. But many are NOT! These example provide one of the largest source of FPs for all the major domain/URI blacklists. But the better domain/URI blacklists have good mechanisms in place to (a) PREVENT... MANY of these from ever becoming FPs in the first place, and (b) and where those mechanism failed, they have good triggers/feedback to remove whitelist such FPs VERY QUICKLY if/when they do occur. In contrast, many who might go overboard by outright blocking on newness... and/or scoring too agressively on newness... may find too-high FP problems kicking their butts in the long run. And when such a FP starts happening, they may not have the proper telemetry to catch/fix it until AFTER much FP damage has happened. Personally, I think that the real problem here is that some of the most famous URI/domain blacklists are NOT catching everything and/or NOT catching everything fast enough... combined with many sys admins failing to make use of ALL the good and low-FP URI/domain blacklists... where they 'd see MUCH better results if they were using ALL of the good URI blacklists! ...but I'm a little biased on this point! :) -- Rob McEwen +1 (478) 475-9032
Re: Domain ages (was Re: SPAM from a registrar)
On Mon, 9 Jun 2014 11:51:21 -0700 (PDT) John Hardin jhar...@impsec.org wrote: So there is merit in building a distributed look-up system using SA. Distributed lookup of *what*, though? Can you clarify that part of your idea? Are you referring to distributed whois queries for a domain name, to determine its age? Well, here's how it could be done. Imagine someone runs a DNS zone for newdomain.example.net. You want to see if example.org is a new domain, so you look up a TXT record for example.org.newdomain.example.net. The DNS software that serves the zone newdomain.example.net runs the following pseudo-code when example.org is looked up: IF example.org is in my database THEN return the TXT record associated with example.org update the last-looked-up time for example.org ELSE generate a TXT record of the form MMDDHHMMSS corresponding to current time (UTC) insert it in the database return it ENDIF A background job will periodically clean out domains that haven't been queried in a long time. The clever part is that once lots of sites begin using this in their SA setups, we'll very quickly build up quite an accurate database of newly-seen domains that's completely independent of any registrar for a data source. Yes, spammers can poison it by specifically looking up a domain, waiting a couple of days, and then spamming. But I think most won't bother (witness how effective greylisting still is.) Furthermore, you can ignore all but the first few hundred lookups before you enter the TXT record in the database; this will make it more expensive for spammers to poison the data. Or you could not enter a record in the database until it has been looked up from 100 different IP addresses... I can think of a few other countermeasures. So who's volunteering to do this? :) Regards, David.
Re: Domain ages (was Re: SPAM from a registrar)
On 6/9/2014 2:51 PM, John Hardin wrote: On Mon, 9 Jun 2014, Kevin A. McGrail wrote: So there is merit in building a distributed look-up system using SA. Distributed lookup of *what*, though? Can you clarify that part of your idea? Are you referring to distributed whois queries for a domain name, to determine its age? Yes. Because whois data is hard to get and many whois servers limit lookups, distributing and sharing the lookup load to determine age of domains IMO has merit.
Re: Domain ages (was Re: SPAM from a registrar)
On 6/9/2014 3:02 PM, Rob McEwen wrote: Domain age is a good metric to factor in. But I'm always fascinated with some people's desire to block all messages with extremely new domains. (NOT saying that this applies to everyone who posted on this thread!) Keep in mind that many large and famous businesses... who have fairly good mail sending practices... sometimes launch a new products complete with links to very newly registered domains. Same is often true for advertisments for things like rock concerts, etc. Or web sites that deal with specific events or hot-topic political issues that appeared out of nowhere. Yes, some of these are UBE. But many are NOT! These example provide one of the largest source of FPs for all the major domain/URI blacklists. But the better domain/URI blacklists have good mechanisms in place to (a) PREVENT... MANY of these from ever becoming FPs in the first place, and (b) and where those mechanism failed, they have good triggers/feedback to remove whitelist such FPs VERY QUICKLY if/when they do occur. In contrast, many who might go overboard by outright blocking on newness... and/or scoring too agressively on newness... may find too-high FP problems kicking their butts in the long run. And when such a FP starts happening, they may not have the proper telemetry to catch/fix it until AFTER much FP damage has happened. Personally, I think that the real problem here is that some of the most famous URI/domain blacklists are NOT catching everything and/or NOT catching everything fast enough... combined with many sys admins failing to make use of ALL the good and low-FP URI/domain blacklists... where they 'd see MUCH better results if they were using ALL of the good URI blacklists! ...but I'm a little biased on this point! :) A great point. My goal is simply to build a system to identify the age of domains and use it as YAIOS or yet another indicator of spamminess not as a poison pill.
Re: Domain ages (was Re: SPAM from a registrar)
Quoting David F. Skoll d...@roaringpenguin.com: On Mon, 9 Jun 2014 11:51:21 -0700 (PDT) John Hardin jhar...@impsec.org wrote: So there is merit in building a distributed look-up system using SA. Distributed lookup of *what*, though? Can you clarify that part of your idea? Are you referring to distributed whois queries for a domain name, to determine its age? Well, here's how it could be done. Imagine someone runs a DNS zone for newdomain.example.net. You want to see if example.org is a new domain, so you look up a TXT record for example.org.newdomain.example.net. The DNS software that serves the zone newdomain.example.net runs the following pseudo-code when example.org is looked up: IF example.org is in my database THEN return the TXT record associated with example.org update the last-looked-up time for example.org ELSE generate a TXT record of the form MMDDHHMMSS corresponding to current time (UTC) insert it in the database return it ENDIF A background job will periodically clean out domains that haven't been queried in a long time. The clever part is that once lots of sites begin using this in their SA setups, we'll very quickly build up quite an accurate database of newly-seen domains that's completely independent of any registrar for a data source. Yes, spammers can poison it by specifically looking up a domain, waiting a couple of days, and then spamming. But I think most won't bother (witness how effective greylisting still is.) Furthermore, you can ignore all but the first few hundred lookups before you enter the TXT record in the database; this will make it more expensive for spammers to poison the data. Or you could not enter a record in the database until it has been looked up from 100 different IP addresses... I can think of a few other countermeasures. So who's volunteering to do this? :) Regards, David. The point was, I have already done this, and have it in production. I did this cause this subject keeps coming up from time to time, and I was personally interested to see the results of it. And I do agree with Rob McEwen on many points. And I would be hisentant to outright block. But so far, and I doubt much in real usage, and haven't found any yet, any issues with blocking 1day outright. But then the only way to be completely sure of that, will be time.
Re: Domain ages (was Re: SPAM from a registrar)
On Mon, 9 Jun 2014, David F. Skoll wrote: On Mon, 9 Jun 2014 11:51:21 -0700 (PDT) John Hardin jhar...@impsec.org wrote: So there is merit in building a distributed look-up system using SA. Distributed lookup of *what*, though? Can you clarify that part of your idea? Are you referring to distributed whois queries for a domain name, to determine its age? The clever part is that once lots of sites begin using this in their SA setups, we'll very quickly build up quite an accurate database of newly-seen domains that's completely independent of any registrar for a data source. Ah, ok, that's where I was confused. The proposal is for a distributed network gathering newly-SEEN domain names, rather than newly-REGISTERED domain names. Thanks for the clarification. I was focusing on the latter. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- You can't reason a person out of a position if he didn't use reason to get there in the first place. -- Kristopher, at Marko's --- 739 days since the first successful private support mission to ISS (SpaceX)
Re: Domain ages (was Re: SPAM from a registrar)
On Mon, 09 Jun 2014 15:24:29 -0400 Patrick Domack patric...@patrickdk.com wrote: The point was, I have already done this, and have it in production. I did this cause this subject keeps coming up from time to time, and I was personally interested to see the results of it. Interesting. If you don't mind my asking... how much data do you collect? How many lookups/day? I was thinking a system that gets lookups from thousands or more SA installations would get a pretty good overview of new domains. A local installation would necessarily see a limited subset. And I do agree with Rob McEwen on many points. And I would be hisentant to outright block. But so far, and I doubt much in real usage, and haven't found any yet, any issues with blocking 1day outright. Or even just holding the mail for a day or so and then re-analyzing it. Regards, David.
Re: Domain ages (was Re: SPAM from a registrar)
On 6/9/2014 3:24 PM, Patrick Domack wrote: The point was, I have already done this, and have it in production. I did this cause this subject keeps coming up from time to time, and I was personally interested to see the results of it. And I do agree with Rob McEwen on many points. And I would be hisentant to outright block. But so far, and I doubt much in real usage, and haven't found any yet, any issues with blocking 1day outright. But then the only way to be completely sure of that, will be time. My conjecture is that many people have built this for lower volume. But you can't be doing much volume or your IP gets blocked from whois servers. The twist I want to do is bring more data back centralized from SA installations such as whois data where it can only be done in a distributed manner. regards, KAM
RE: Domain ages (was Re: SPAM from a registrar)
If SEM was able to detect newly registered domains more quickly then that would solve the problem. From: John Hardin jhar...@impsec.org Sent: Monday, June 09, 2014 2:24 PM To: users@spamassassin.apache.org Subject: Re: Domain ages (was Re: SPAM from a registrar) On Mon, 9 Jun 2014, David F. Skoll wrote: On Mon, 9 Jun 2014 11:51:21 -0700 (PDT) John Hardin jhar...@impsec.org wrote: So there is merit in building a distributed look-up system using SA. Distributed lookup of *what*, though? Can you clarify that part of your idea? Are you referring to distributed whois queries for a domain name, to determine its age? The clever part is that once lots of sites begin using this in their SA setups, we'll very quickly build up quite an accurate database of newly-seen domains that's completely independent of any registrar for a data source. Ah, ok, that's where I was confused. The proposal is for a distributed network gathering newly-SEEN domain names, rather than newly-REGISTERED domain names. Thanks for the clarification. I was focusing on the latter. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- You can't reason a person out of a position if he didn't use reason to get there in the first place. -- Kristopher, at Marko's --- 739 days since the first successful private support mission to ISS (SpaceX)
Re: Domain ages (was Re: SPAM from a registrar)
On Mon, 9 Jun 2014, Kevin A. McGrail wrote: On 6/9/2014 2:51 PM, John Hardin wrote: On Mon, 9 Jun 2014, Kevin A. McGrail wrote: So there is merit in building a distributed look-up system using SA. Distributed lookup of *what*, though? Can you clarify that part of your idea? Are you referring to distributed whois queries for a domain name, to determine its age? Yes. Because whois data is hard to get and many whois servers limit lookups, distributing and sharing the lookup load to determine age of domains IMO has merit. Ah, I think there's still two different assumptions occurring in this discussion: newly-seen (David and Patrick) vs. newly-registered (me and Kevin)... Maybe we need to clarify that first. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- You can't reason a person out of a position if he didn't use reason to get there in the first place. -- Kristopher, at Marko's --- 739 days since the first successful private support mission to ISS (SpaceX)
Re: Domain ages (was Re: SPAM from a registrar)
On 6/9/2014 3:33 PM, John Hardin wrote: On Mon, 9 Jun 2014, Kevin A. McGrail wrote: On 6/9/2014 2:51 PM, John Hardin wrote: On Mon, 9 Jun 2014, Kevin A. McGrail wrote: So there is merit in building a distributed look-up system using SA. Distributed lookup of *what*, though? Can you clarify that part of your idea? Are you referring to distributed whois queries for a domain name, to determine its age? Yes. Because whois data is hard to get and many whois servers limit lookups, distributing and sharing the lookup load to determine age of domains IMO has merit. Ah, I think there's still two different assumptions occurring in this discussion: newly-seen (David and Patrick) vs. newly-registered (me and Kevin)... Maybe we need to clarify that first. Good clarification. The spam I envision stopping is spammers using things like stolen credit cards or trial accounts to register domains that they then spam and then disappear quite quickly. So this builds a database of domain whois data (initial discussions focused on the creation date) using distributed SA nodes to build the data. And I chose to discuss it here because I get more ideas than I have time and resources to implement. Regards, KAM
Re: Domain ages (was Re: SPAM from a registrar)
On 6/9/2014 3:31 PM, David Jones wrote: If SEM was able to detect newly registered domains more quickly then that would solve the problem. That is the crux of the issue, yes. So how do you identify new domains if the registrars/registries won't give you the data? That's the problem my idea solves by monitoring newly seen domains with the idea being that spammers are not going to buy domains and sit on them before using them. Regards, KAM
RE: Domain ages (was Re: SPAM from a registrar)
On Mon, 9 Jun 2014, David Jones wrote: If SEM was able to detect newly registered domains more quickly then that would solve the problem. Oh, agreed. The problem is, a registrar feed of registration changes costs a lot, and this is a free project. That's why I suggested trying to develop relationships with registrars, to maybe get them onboard with providing this data for free for this purpose. It's possible that the Apache name could provide cachet to get registars onboard to provide rsync'able data feeds of domain names registered in the last N days. It might be possible/better to get them to provide the data to URIBL.org (to act as an aggregator) with a license to provide the data free via DNS (i.e. non-bulk access) and at a nominal fee for rsync access (which URIBL already charges for the data they collect). -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- You can't reason a person out of a position if he didn't use reason to get there in the first place. -- Kristopher, at Marko's --- 739 days since the first successful private support mission to ISS (SpaceX)
Re: Domain ages (was Re: SPAM from a registrar)
On 06/09/2014 09:38 PM, Kevin A. McGrail wrote: That is the crux of the issue, yes. So how do you identify new domains if the registrars/registries won't give you the data? That's the problem my idea solves by monitoring newly seen domains with the idea being that spammers are not going to buy domains and sit on them before using them. You get the TLD zone files... and depending on your budget you get them once/24hrs or hourly diffs (if you can affford a house in The Hamptons, you can afford the diffs .-) Some TLDs won't handout zone, period.
Re: Domain ages (was Re: SPAM from a registrar)
On Mon, Jun 9, 2014 at 8:43 PM, Kevin A. McGrail kmcgr...@pccc.com wrote: I think the core issue is that age of domains is a good indicator of spam. So there is merit in building a distributed look-up system using SA. I have more ideas than resources, of course... I repeat my question: which domain? HELO, MAIL FROM, From:, ...? -- Matthias
Re: Domain ages (was Re: SPAM from a registrar)
On 6/9/2014 4:25 PM, Matthias Leisi wrote: On Mon, Jun 9, 2014 at 8:43 PM, Kevin A. McGrail kmcgr...@pccc.com mailto:kmcgr...@pccc.com wrote: I think the core issue is that age of domains is a good indicator of spam. So there is merit in building a distributed look-up system using SA. I have more ideas than resources, of course... I repeat my question: which domain? HELO, MAIL FROM, From:, ...? I envision it for potentially any and all domains in the email.
Re: Domain ages (was Re: SPAM from a registrar)
On Mon, Jun 9, 2014 at 9:11 PM, David F. Skoll d...@roaringpenguin.com wrote: The clever part is that once lots of sites begin using this in their SA setups, we'll very quickly build up quite an accurate database of newly-seen domains that's completely independent of any registrar for a data source. dnswl.org (and many other DNSxLs) already have some of that data as part of their parsing/handling of DNS logs. For Furthermore, you can ignore all but the first few hundred lookups before you enter the TXT record in the database; this will make it more expensive for spammers to poison the data. Or you could not enter a record in the database until it has been looked up from 100 different IP addresses... I can think of a few other countermeasures. So who's volunteering to do this? :) We had some plans to publish such data. However since it is not really clear what domains to look for, we did not pursue that a lot further. We have at least a primary domain for each DNSWL record, but at least historically we were not strict in what type of domain to put there (we like to use the domain name that most closely links the IPs to the administratively responsible owner, which is admittedly somewhat vague). Based on the useage data we gather, we can pretty accurately extract a last seen date for a particular domain (or, it's associated IPs to be exact). *But*, again: which domains would be queried for such a list? -- Matthias
Re: Domain ages (was Re: SPAM from a registrar)
Quoting Matthias Leisi matth...@leisi.net: On Mon, Jun 9, 2014 at 8:43 PM, Kevin A. McGrail kmcgr...@pccc.com wrote: I think the core issue is that age of domains is a good indicator of spam. So there is merit in building a distributed look-up system using SA. I have more ideas than resources, of course... I repeat my question: which domain? HELO, MAIL FROM, From:, ...? -- Matthias HELO hasn't matched anything in my tests. MAIL FROM has matched many, though the helo's are always a different domain From I have only started doing yesterday, and not sure exactly how I will track them. Likely just wait a few days, and check my ham/spam folders and compare what rules where hit.
Re: Domain ages (was Re: SPAM from a registrar)
On 06/09/2014 10:32 PM, Patrick Domack wrote: Quoting Matthias Leisi matth...@leisi.net: On Mon, Jun 9, 2014 at 8:43 PM, Kevin A. McGrail kmcgr...@pccc.com wrote: I think the core issue is that age of domains is a good indicator of spam. So there is merit in building a distributed look-up system using SA. I have more ideas than resources, of course... I repeat my question: which domain? HELO, MAIL FROM, From:, ...? -- Matthias HELO hasn't matched anything in my tests. MAIL FROM has matched many, though the helo's are always a different domain From I have only started doing yesterday, and not sure exactly how I will track them. Likely just wait a few days, and check my ham/spam folders and compare what rules where hit. LOTS of the recent .us .me will match sender/ptr/A/HELO
Re: Domain ages (was Re: SPAM from a registrar)
On Mon, 9 Jun 2014 22:31:55 +0200 Matthias Leisi matth...@leisi.net wrote: *But*, again: which domains would be queried for such a list? I think MAIL FROM domain. Regards, David.
Re: Domain ages (was Re: SPAM from a registrar)
On Mon, June 9, 2014 15:35, Patrick Domack wrote: I guess what would need to be hammered out, is, the exact info wanted. We know age, and registrar. Though doing the registrar isn't so simple, as the same for just ENOM changes between tld, and even within a single tld (likely from the mergers they had). My investigations of the domains used against us revealed that all of the handful checked were between 4 and 20 hours old when first encountered by our servers. It would suffice I think to have a negative lookup RTBL service where if a domain is not listed therein then may be considered as new, at least insofar as mailing traffic is concerned. The registrar and the age of the domain need not concern us overmuch at the outset of a spam attack. What is more important to know is whether the domain has been seen by others before and how long before so that the information in DOB and SEM can be considered in that light. Lookup domains may be added as and when they are encountered albeit after some delay and only if some threshold of volume and distinct number of enquiring hosts is passed. A graded approach is probably called for with one listing a previously unseen domain only after 24 hours from the first enquiry, one only after 48, and so on. Of course, the domains in question need to be verified before being added. And other precautions are no doubt necessary to avoid poisoning or advance loading subversion attempts. Comments? -- *** E-Mail is NOT a SECURE channel *** James B. Byrnemailto:byrn...@harte-lyne.ca Harte Lyne Limited http://www.harte-lyne.ca 9 Brockley Drive vox: +1 905 561 1241 Hamilton, Ontario fax: +1 905 561 0757 Canada L8E 3C3
Re: Domain ages (was Re: SPAM from a registrar)
On Mon, Jun 9, 2014 at 9:11 PM, David F. Skoll d...@roaringpenguin.com wrote: The DNS software that serves the zone newdomain.example.net runs the following pseudo-code when example.org is looked up: [..] So who's volunteering to do this? :) *raises hand* I still have an experimental DNS server (written in Perl) lying around that this more-or-less what is described here. The overall system would need a bit more thought, though. * Distributed over n nodes. Given that data can have pretty long TTL, it does not need a lot of nodes, but still the distributed nature brings some challenges. * Definition of the granularity of data - should a first seen date be returned, or an age (in days?) * Querying whois servers is not practical at that scale. * How would the queries be sent to the nodes? Domain-based BL-type queries? * Would the SA project take on some operational responsibilities? * The dnswl.org project can sponsor resources and take on some operational aspects, but we would welcome some support. -- Matthias
Re: Domain ages (was Re: SPAM from a registrar)
On 06/09/2014 10:43 PM, James B. Byrne wrote: On Mon, June 9, 2014 15:35, Patrick Domack wrote: I guess what would need to be hammered out, is, the exact info wanted. We know age, and registrar. Though doing the registrar isn't so simple, as the same for just ENOM changes between tld, and even within a single tld (likely from the mergers they had). My investigations of the domains used against us revealed that all of the handful checked were between 4 and 20 hours old when first encountered by our servers. It would suffice I think to have a negative lookup RTBL service where if a domain is not listed therein then may be considered as new, at least insofar as mailing traffic is concerned. The registrar and the age of the domain need not concern us overmuch at the outset of a spam attack. What is more important to know is whether the domain has been seen by others before and how long before so that the information in DOB and SEM can be considered in that light. Lookup domains may be added as and when they are encountered albeit after some delay and only if some threshold of volume and distinct number of enquiring hosts is passed. A graded approach is probably called for with one listing a previously unseen domain only after 24 hours from the first enquiry, one only after 48, and so on. Of course, the domains in question need to be verified before being added. And other precautions are no doubt necessary to avoid poisoning or advance loading subversion attempts. Comments? You have a domain reputation method on your drawing board and imo, has some flaws: - Delayed data is good for research, not to efficiently stop spam. - Verifying anything that large needs 40k indians in the basement or huge clusters of cycles doing something - neither is trivial or cheap. - There's a bunch of Passsive DNS projects which do what you're describing and non will work as the FUSSP - they're datapoints which can be combined wiht other stuff to achieve something (aka research)
Re: Domain ages (was Re: SPAM from a registrar)
On 06/09/2014 12:29 PM, Kevin A. McGrail wrote: On 6/9/2014 3:24 PM, Patrick Domack wrote: The point was, I have already done this, and have it in production. I did this cause this subject keeps coming up from time to time, and I was personally interested to see the results of it. And I do agree with Rob McEwen on many points. And I would be hisentant to outright block. But so far, and I doubt much in real usage, and haven't found any yet, any issues with blocking 1day outright. But then the only way to be completely sure of that, will be time. My conjecture is that many people have built this for lower volume. But you can't be doing much volume or your IP gets blocked from whois servers. The twist I want to do is bring more data back centralized from SA installations such as whois data where it can only be done in a distributed manner. regards, KAM A caching whois client (jwhois, for example) can significantly reduce the volume of queries.
Re: Domain ages (was Re: SPAM from a registrar)
On Mon, Jun 9, 2014 at 11:31 PM, Richard Doyle lists...@islandnetworks.com wrote: A caching whois client (jwhois, for example) can significantly reduce the volume of queries. You will need to query potentially hundreds or thousands of domains *per day* - mostly throw away domains from spammers. 1) What are the typical rate limits on public whois servers? 2) How to protect against attackers sending random non-existant domain names your way, thus ensuring you hit rate limites early? 3) How to parse the myriads of formats sent by whois servers? 4) How do you handle TLDs which do not publish registration dates, like eg .de? (At least they did not last time I checked.) Whois is not a feasible data source. -- Matthias
Re: Domain ages (was Re: SPAM from a registrar)
Quoting Matthias Leisi matth...@leisi.net: On Mon, Jun 9, 2014 at 11:31 PM, Richard Doyle lists...@islandnetworks.com wrote: A caching whois client (jwhois, for example) can significantly reduce the volume of queries. You will need to query potentially hundreds or thousands of domains *per day* - mostly throw away domains from spammers. 1) What are the typical rate limits on public whois servers? 2) How to protect against attackers sending random non-existant domain names your way, thus ensuring you hit rate limites early? 3) How to parse the myriads of formats sent by whois servers? 4) How do you handle TLDs which do not publish registration dates, like eg .de? (At least they did not last time I checked.) Whois is not a feasible data source. -- Matthias 1) I dunno, but I am doing around 15k lookups a day, from a single ip, without getting limited/blocked 2) This is hard, and I don't know, currently the postfix reject unknown sender helps solve this for me, but won't for dns based lookups 3) This, while annoying, is solved in my code, not too hard 4) These I just don't bother doing lookups for, there is no solution, other than to let them bypass this system, or rate them via seen before method.
Re: Domain ages (was Re: SPAM from a registrar)
On 06/09/2014 02:42 PM, Matthias Leisi wrote: On Mon, Jun 9, 2014 at 11:31 PM, Richard Doyle lists...@islandnetworks.com mailto:lists...@islandnetworks.com wrote: A caching whois client (jwhois, for example) can significantly reduce the volume of queries. You will need to query potentially hundreds or thousands of domains *per day* - mostly throw away domains from spammers. 1) What are the typical rate limits on public whois servers? Apparently higher than my usage (cached names aren't rechecked) 2) How to protect against attackers sending random non-existant domain names your way, thus ensuring you hit rate limites early? Sender verification 3) How to parse the myriads of formats sent by whois servers? Don't try (see 4) 4) How do you handle TLDs which do not publish registration dates, like eg .de? (At least they did not last time I checked.) I only check .com, .net and .org Whois is not a feasible data source. Whois certainly has limited usefulness, but is a feasible data source within those limits -- Matthias -Richard
