On Fri, May 13, 2016 at 07:59:57PM -0400, Ciro Iriarte wrote: > Out of curiosity, how does this part of the platform work?:
Hi Ciro, In general, I don't want to spam the pdns-users people with information about the PowerDNS Platform, as outlined on https://www.powerdns.com/platform.html since it is not part of the open source stuff, so most of the community won't have any use for the information. (the story of our non-open source work is on https://blog.powerdns.com/2016/02/23/an-important-update-on-new-powerdns-products/ ) But let me explain how this hangs together and what is part of the open source version. The PowerDNS Recursor 4.0.0 and dnsdist 1.0.0 have the ability to emit a stream of protobufs over TCP/IP. In case of the Recursor, this also has the 'policy reason' why a request was intercepted by the RPZ module. To receive that stream, use something like xinetd to listen on a TCP/IP port and store the data to a file. It can then be processed by any tool that can understand Protobuf. The schema is here: https://github.com/PowerDNS/pdns/blob/master/pdns/dnsmessage.proto In the very near future this will also be able to emit standard dnstap messages. > Long term full query logging & rapid searching > > - Dimensioned at a trillion queries/day (1000 billion) on commodity > hardware with long term retention > - For security research, lawful intercept/data retention requirements, > customer intelligence, quality assurance/diagnostics This describes our protobuf receiver 'dstore' which through some clever programming techniques can store trillions of DNS messages and serve them up again reasonably quickly. It is not a generic database, but it is really fast and nearly maintenance free and has no further dependencies (so you don't need to be a "big data engineer" to benefit from it). This can be very useful to investigate customer complaints of DNS slowness, or that a domain was down etc. It is also extremely powerful for finding infected users. A commandline like: $ dgrep t=week pr=spamhaus-dbl | jq ".items[].origRequestor" | sort | uniq -c \ | sort -rn | head -10 .. will find in a few seconds the top-10 IP addresses that over the past week had the most queries blocked by the 'spamhaus-dbl' RPZ. The output of dgrep is JSON, easily queried and selected by jq. But again - I don't want to promote our commercial Platform offering here too much. For the open source world, you should be able to bake up a solution based on elastic search, kibana etc that ingests our protobufs. Bert _______________________________________________ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users
