Re: Cooperative data gathering project.
On Sat, 19 Dec 2009, Jason Haar wrote: On 12/19/2009 04:51 AM, Jonas Eckerman wrote: (And if more security is needed the easiest way would be to simple limit access to approved IP addresses.) Except that a token would enable one "owner" with multiple SA instances on separate networks to come across as one entity - that could be desirable too. It all depends on what you are trying to achieve of course. Also UDP means forgery is a bigger risk - so IP-based checks are less reliable. Right. You'd need to include an id/auth token in the UDP packet. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- "Bother," said Pooh as he struggled with /etc/sendmail.cf, "it never does quite what I want. I wish Christopher Robin was here." -- Peter da Silva in a.s.r --- 7 days until Christmas
Re: Cooperative data gathering project.
On 12/19/2009 04:51 AM, Jonas Eckerman wrote: > > (And if more security is needed the easiest way would be to simple > limit access to approved IP addresses.) Except that a token would enable one "owner" with multiple SA instances on separate networks to come across as one entity - that could be desirable too. It all depends on what you are trying to achieve of course. Also UDP means forgery is a bigger risk - so IP-based checks are less reliable. -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: Cooperative data gathering project.
Per Jessen wrote: DNS lookups are usually tried done with UDP first, Sure, DNS usually uses UDP, but the DNS resolver also waits for an answer, wich is simply a waste of time when the sender doesn't need the answer. Add to this that resolving one address may result in multiple queries and that a DNS answer often containes more that the queried info and you get more overhead. > but I agree, just use UDP. Absolutely. Imo, the approach suggested by Marc is a text-book example of when to use UDP. (And if more security is needed the easiest way would be to simple limit access to approved IP addresses.) Regards /Jonas -- Jonas Eckerman Fruktträdet & Förbundet Sveriges Dövblinda http://www.fsdb.org/ http://www.frukt.org/ http://whatever.frukt.org/
Re: Cooperative data gathering project.
Jason Haar wrote: Then the third filed is NONE. That's how I do it. But the idea is that any kind of daya can be collectively gathered and distributed. Instead of a TCP channel (which means software), what about using DNS? If the SA clients did RBL lookups that contained the details as part of the query, With any sane SpamAssassin setup for multiple users this wouldn't work. Any SA install except for very small mail flows should use a caching DNS server/proxy, preferably one that caches negative results. It's also a good idea if the caching server used for DNSL checks enforces a minimum TTL. This results in repeated queries not making it to the origin servers. Even if the origin server uses ridicilously low TTLs. The distributed caching nature of DNS is a reason why DNSLs are so efficient, but also one reason why DNS isn't suitable for everything. Regards /Jonas -- Jonas Eckerman Fruktträdet & Förbundet Sveriges Dövblinda http://www.fsdb.org/ http://www.frukt.org/ http://whatever.frukt.org/
Re: Cooperative data gathering project.
Marc Perkel wrote: spam 1.2.3.4 example.com ham 5.6.7.8 example2.com Sending these one line TCP messages if fairly easy. Why use TCP for this? Establishing a connection channel for simple short mesages where a return code is not required introduces pointless overhead. It'd be much simpler using UDP instead. Regards /Jonas -- Jonas Eckerman Fruktträdet & Förbundet Sveriges Dövblinda http://www.fsdb.org/ http://www.frukt.org/ http://whatever.frukt.org/
Re: Cooperative data gathering project.
Henrik K wrote: > Ok, while DNS would "allow" that, it would be a real waste of a > protocol. Why would you want to make the sending party wait for a > response that only adds delays and has no purpose? Simply send a UDP > packet and be done with it. No TCP or DNS overhead. One or two lines > of perl. DNS lookups are usually tried done with UDP first, but I agree, just use UDP. /Per Jessen, Zürich
Re: Cooperative data gathering project.
On Fri, Dec 18, 2009 at 08:20:47AM +1300, Jason Haar wrote: > On 12/18/2009 05:31 AM, Marc Perkel wrote: > > > > In this case the idea is to gather data in real time. So those who > > gather data need to be able to send the data to a central place that > > receives the data and then makes it available to everyone. > > > You've lost me there - DNS would allow you to do that. You run the DNS > server, SA does the queries, they get routed through the standard DNS > hierarchy, you get to see them (and respond), interpret and make that > data available via any mechanism you want - probably also via (a > different?) DNS. A short TTL would reduce lost data (in fact, the only > possible cached-caused data loss would be for duplicate queries from the > same SA instance) Ok, while DNS would "allow" that, it would be a real waste of a protocol. Why would you want to make the sending party wait for a response that only adds delays and has no purpose? Simply send a UDP packet and be done with it. No TCP or DNS overhead. One or two lines of perl.
Re: Cooperative data gathering project.
On 12/18/2009 05:31 AM, Marc Perkel wrote: > > In this case the idea is to gather data in real time. So those who > gather data need to be able to send the data to a central place that > receives the data and then makes it available to everyone. > You've lost me there - DNS would allow you to do that. You run the DNS server, SA does the queries, they get routed through the standard DNS hierarchy, you get to see them (and respond), interpret and make that data available via any mechanism you want - probably also via (a different?) DNS. A short TTL would reduce lost data (in fact, the only possible cached-caused data loss would be for duplicate queries from the same SA instance) -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: Cooperative data gathering project.
Jason Haar wrote: On 12/17/2009 03:30 PM, Marc Perkel wrote: Then the third filed is NONE. That's how I do it. But the idea is that any kind of daya can be collectively gathered and distributed. Instead of a TCP channel (which means software), what about using DNS? If the SA clients did RBL lookups that contained the details as part of the query, then if your end parses DNS logs (I'm thinking djbdns, don't know about BIND), then you could extract the data yourself. You could even introduce a token into the RBL to stop the bad guys corrupting your corpus (a problem you'll have to deal with anyway whatever the network mechanism). e.g. (token == "834ufg754") spam.1.2.3.4.834ufg754.newrbl.com ham.5.6.7.8.834ufg754.newrbl.com ie only the dns logs that contain valid tokens are legitimate In this case the idea is to gather data in real time. So those who gather data need to be able to send the data to a central place that receives the data and then makes it available to everyone.
Re: Cooperative data gathering project.
Jason Haar wrote: > On 12/17/2009 03:30 PM, Marc Perkel wrote: >> Then the third filed is NONE. That's how I do it. But the idea is >> that any kind of daya can be collectively gathered and distributed. >> > Instead of a TCP channel (which means software), what about using DNS? > If the SA clients did RBL lookups that contained the details as part > of the query, then if your end parses DNS logs (I'm thinking djbdns, > don't know about BIND), then you could extract the data yourself. There is significant chance of queries not making it to Marc's DNS due to caching. /Per Jessen, Zürich
Re: Cooperative data gathering project.
On 12/17/2009 03:30 PM, Marc Perkel wrote: Then the third filed is NONE. That's how I do it. But the idea is that any kind of daya can be collectively gathered and distributed. Instead of a TCP channel (which means software), what about using DNS? If the SA clients did RBL lookups that contained the details as part of the query, then if your end parses DNS logs (I'm thinking djbdns, don't know about BIND), then you could extract the data yourself. You could even introduce a token into the RBL to stop the bad guys corrupting your corpus (a problem you'll have to deal with anyway whatever the network mechanism). e.g. (token == "834ufg754") spam.1.2.3.4.834ufg754.newrbl.com ham.5.6.7.8.834ufg754.newrbl.com ie only the dns logs that contain valid tokens are legitimate -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: Cooperative data gathering project.
Then the third filed is NONE. That's how I do it. But the idea is that any kind of daya can be collectively gathered and distributed. R-Elists wrote: marc, what if there is no RDNS ? ;-) - rh
RE: Cooperative data gathering project.
marc, what if there is no RDNS ? ;-) - rh
Re: Cooperative data gathering project.
Mike Cardwell wrote: On 16/12/2009 15:37, Marc Perkel wrote: I have an idea for a cooperative data gathering project and it's similar to what we at Junk Email Filter are doing internally. The idea to that a number of you on the list who run a large spam filtering operation send one line messages reporting IP addresses with a key word like spam, ham, or other useful messages that we might want to gather information about. Data might look like this: spam 1.2.3.4 example.com ham 5.6.7.8 example2.com What is "example.com" ? The envelope sender domain? The PTR? The From header domain? The RDNS of the IP, although I could look that up myself and include it in the processed data. -- Marc Perkel - Sales/Support supp...@junkemailfilter.com http://www.junkemailfilter.com Junk Email Filter dot com 415-992-3400
Re: Cooperative data gathering project.
On 16/12/2009 15:37, Marc Perkel wrote: > I have an idea for a cooperative data gathering project and it's similar > to what we at Junk Email Filter are doing internally. The idea to that a > number of you on the list who run a large spam filtering operation send > one line messages reporting IP addresses with a key word like spam, ham, > or other useful messages that we might want to gather information about. > Data might look like this: > > spam 1.2.3.4 example.com > ham 5.6.7.8 example2.com What is "example.com" ? The envelope sender domain? The PTR? The From header domain? -- Mike Cardwell - IT Consultant and LAMP developer Cardwell IT Ltd. (UK Reg'd Company #06920226) http://cardwellit.com/ Technical Blog: https://secure.grepular.com/blog/
Cooperative data gathering project.
I have an idea for a cooperative data gathering project and it's similar to what we at Junk Email Filter are doing internally. The idea to that a number of you on the list who run a large spam filtering operation send one line messages reporting IP addresses with a key word like spam, ham, or other useful messages that we might want to gather information about. Data might look like this: spam 1.2.3.4 example.com ham 5.6.7.8 example2.com I'll set up some servers to receive these one line messages, gather up the data, and make it available to all who participate, or maybe just make it public. Sending these one line TCP messages if fairly easy. If you are running Exim you can use the readsocket command to send the message. If you can run an external script you can send it with netcat. I'm sure doing it on perl is trivial. We'll pick an IP address and a secret port to send the data to. On my send a one line server is created running SOCAT that just receives these lines and appends them to a text file. These lines will be gathered up into bigger files and available to be downloaded and used to create various white, black, yellow, or other DNS lists. This is the system I use internally and it works really well. So I'm thinking it would be useful on a larger scale. Who is interested?
