Re: [pmacct-discussion] HTTP traffic classification
On 03/24/2014 08:14:25 AM, Chris Wilson wrote: > I'd like to see the *content* of DNS requests and responses available > to > be logged in data records by pmacct. It can be very helpful in > identifying > which website someone was trying to access, when all we have is an IP > address. I accept that not everybody would want this, but I do. I see. That makes sense. I misunderstood what you were trying to do. I was thinking http. Karl Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] HTTP traffic classification
Hi Karl, On Mon, 24 Mar 2014, Karl O. Pinc wrote: On 03/24/2014 06:31:30 AM, Stathis Gkotsis wrote: Concerning HTTP: I guess the thing to output would be hostname, since you can have multiple HTTP requests to different URLs inside one TCP Session.About DNS, what should be outputted? I guess the hostname for A queries is good enough to start with. I'm not clear on where DNS would fit into this. Offhand, DNS lookups (and then reverse DNS lookups, etc.) should not be part of pmacct. There's just too much latency. People who want that sort of thing should work out how to do it outside of pmacct. I'd like to see the *content* of DNS requests and responses available to be logged in data records by pmacct. It can be very helpful in identifying which website someone was trying to access, when all we have is an IP address. I accept that not everybody would want this, but I do. Cheers, Chris. -- Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838 Citylife House, Sturton Street, Cambridge, CB1 2QF, UK Aptivate is a not-for-profit company registered in England and Wales with company number 04980791. ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] HTTP traffic classification
On 03/24/2014 06:31:30 AM, Stathis Gkotsis wrote: > Hi all, > Concerning HTTP: I guess the thing to output would be hostname, since > you can have multiple HTTP requests to different URLs inside one TCP > Session.About DNS, what should be outputted? I guess the hostname for > A queries is good enough to start with. I'm not clear on where DNS would fit into this. Offhand, DNS lookups (and then reverse DNS lookups, etc.) should not be part of pmacct. There's just too much latency. People who want that sort of thing should work out how to do it outside of pmacct. Karl Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] HTTP traffic classification
Hi all, Concerning HTTP: I guess the thing to output would be hostname, since you can have multiple HTTP requests to different URLs inside one TCP Session.About DNS, what should be outputted? I guess the hostname for A queries is good enough to start with. BR,Stathis > Date: Sun, 23 Mar 2014 13:56:32 + > From: pa...@pmacct.net > To: pmacct-discussion@pmacct.net > Subject: Re: [pmacct-discussion] HTTP traffic classification > > Dears, > > First off, interesting discussion. Under the assumption we speak > libpcap and not NetFlow/IPFIX, I confirm, as it was already clear > from Slava and Chris emails, that there is nothing built-in to do > this in pmacct. > > I see two possible avenues for this: a) go the classification way, > ie. most probably write a binary classifier (*) since regex would > not help with binary protocols (i've read DNS also) and is overall > pretty limiting. A new primitive, of type string, should be defined > to contain, say, URLs or DNS data. b) Slightly expand and leverage > the aggregate_primitives framework, active for libpcap and NetFlow/ > IPFIX. See at this propo "examples/primitives.lst" (last couple of > examples) in the pmacct distribution tarball. The part to expand is > the support for variable-length jumps. > > I'm happy to support on this (so for example to facilitate where to > start, how to make things consistent to the rest, etc.) but somebody > has to take the actual development, which is non trivial but does not > look like crazy amount either, on him - and hopefully contribute it > back to the community. > > Look forward at your thoughts. > > Cheers, > Paolo > > (*) http://www.pmacct.net/classification/pmacct-classifiers-20060321.tar.gz > > On Sat, Mar 22, 2014 at 08:18:01PM +, Chris Wilson wrote: > > Hi all, > > > > On Sat, 22 Mar 2014, Viacheslav Dubrovskyi wrote: > > >22.03.2014 21:20, Stathis Gkotsis пишет: > > >>First, I would like to thank you for the great product, pmacct > > >>has proven very useful to me, which brings me to my question :) > > >>I see that it is possible to enable traffic classification, > > >>which is about detecting L7 protocol. I am particularly > > >>interested in HTTP and also outputting the hostname or url, e.g. > > >>in exports via the print module. Is this somehow possible? > > > > > >IMHO better use special tools https://github.com/jbittel/httpry > > > > I'm also interested in this. Even if it's captured by a separate > > tool (and I'm not sure why it couldn't be integrated with pmacct's > > L7 classifiers) I would really like to be able to log http and https > > hostnames of connections, and correlate them with flows recorded by > > pmacct and DNS requests and responses. > > > > It's not clear that httpry can log the source and destination host > > and port at all, let alone store it in a SQL database (no sample > > output is provided), and presumably it does nothing with https. > > > > Cheers, Chris. > > -- > > Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838 > > Citylife House, Sturton Street, Cambridge, CB1 2QF, UK > > > > Aptivate is a not-for-profit company registered in England and Wales > > with company number 04980791. > > > ___ > > pmacct-discussion mailing list > > http://www.pmacct.net/#mailinglists > > > ___ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] HTTP traffic classification
Dears, First off, interesting discussion. Under the assumption we speak libpcap and not NetFlow/IPFIX, I confirm, as it was already clear from Slava and Chris emails, that there is nothing built-in to do this in pmacct. I see two possible avenues for this: a) go the classification way, ie. most probably write a binary classifier (*) since regex would not help with binary protocols (i've read DNS also) and is overall pretty limiting. A new primitive, of type string, should be defined to contain, say, URLs or DNS data. b) Slightly expand and leverage the aggregate_primitives framework, active for libpcap and NetFlow/ IPFIX. See at this propo "examples/primitives.lst" (last couple of examples) in the pmacct distribution tarball. The part to expand is the support for variable-length jumps. I'm happy to support on this (so for example to facilitate where to start, how to make things consistent to the rest, etc.) but somebody has to take the actual development, which is non trivial but does not look like crazy amount either, on him - and hopefully contribute it back to the community. Look forward at your thoughts. Cheers, Paolo (*) http://www.pmacct.net/classification/pmacct-classifiers-20060321.tar.gz On Sat, Mar 22, 2014 at 08:18:01PM +, Chris Wilson wrote: > Hi all, > > On Sat, 22 Mar 2014, Viacheslav Dubrovskyi wrote: > >22.03.2014 21:20, Stathis Gkotsis пишет: > >>First, I would like to thank you for the great product, pmacct > >>has proven very useful to me, which brings me to my question :) > >>I see that it is possible to enable traffic classification, > >>which is about detecting L7 protocol. I am particularly > >>interested in HTTP and also outputting the hostname or url, e.g. > >>in exports via the print module. Is this somehow possible? > > > >IMHO better use special tools https://github.com/jbittel/httpry > > I'm also interested in this. Even if it's captured by a separate > tool (and I'm not sure why it couldn't be integrated with pmacct's > L7 classifiers) I would really like to be able to log http and https > hostnames of connections, and correlate them with flows recorded by > pmacct and DNS requests and responses. > > It's not clear that httpry can log the source and destination host > and port at all, let alone store it in a SQL database (no sample > output is provided), and presumably it does nothing with https. > > Cheers, Chris. > -- > Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838 > Citylife House, Sturton Street, Cambridge, CB1 2QF, UK > > Aptivate is a not-for-profit company registered in England and Wales > with company number 04980791. > ___ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] HTTP traffic classification
Hi all, On Sat, 22 Mar 2014, Viacheslav Dubrovskyi wrote: 22.03.2014 21:20, Stathis Gkotsis пишет: First, I would like to thank you for the great product, pmacct has proven very useful to me, which brings me to my question :) I see that it is possible to enable traffic classification, which is about detecting L7 protocol. I am particularly interested in HTTP and also outputting the hostname or url, e.g. in exports via the print module. Is this somehow possible? IMHO better use special tools https://github.com/jbittel/httpry I'm also interested in this. Even if it's captured by a separate tool (and I'm not sure why it couldn't be integrated with pmacct's L7 classifiers) I would really like to be able to log http and https hostnames of connections, and correlate them with flows recorded by pmacct and DNS requests and responses. It's not clear that httpry can log the source and destination host and port at all, let alone store it in a SQL database (no sample output is provided), and presumably it does nothing with https. Cheers, Chris. -- Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838 Citylife House, Sturton Street, Cambridge, CB1 2QF, UK Aptivate is a not-for-profit company registered in England and Wales with company number 04980791. ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] HTTP traffic classification
22.03.2014 21:20, Stathis Gkotsis пишет: Hi all, First, I would like to thank you for the great product, pmacct has proven very useful to me, which brings me to my question :) I see that it is possible to enable traffic classification, which is about detecting L7 protocol. I am particularly interested in HTTP and also outputting the hostname or url, e.g. in exports via the print module. Is this somehow possible? IMHO better use special tools https://github.com/jbittel/httpry -- WBR, Viacheslav Dubrovskyi smime.p7s Description: Криптографическая подпись S/MIME ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] HTTP traffic classification
Hi all, First, I would like to thank you for the great product, pmacct has proven very useful to me, which brings me to my question :)I see that it is possible to enable traffic classification, which is about detecting L7 protocol. I am particularly interested in HTTP and also outputting the hostname or url, e.g. in exports via the print module. Is this somehow possible? Best regards,Stathis ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
