Re: [pmacct-discussion] Measurement accuracy issues
Well thanks, but basically darkstat has zero configuration. It just outputs the total download traffic per host IP. And I believe that's the same thing returned by that SQL query I posted. Since both apps are running on the same server, I guess they should pretty much be equivalent. Please do let me know if you think there are specific tests I can do to ensure equivalency I will in the mean time, try the previosuly suggested test of downloading a fixed file size Thanks and best regards On Tue, Jun 10, 2008 at 12:59 PM, alex <[EMAIL PROTECTED]> wrote: > Hello Ahmed, > I only want to add that you must be strong ensure that you compare > same flows of data. It is seem obviously but you can easy miss something > in setting of both programs and compare hasn't any sense. > > Alex > > > > Hi Ahmed, > > > > On Tue, 10 Jun 2008, Ahmed Kamal wrote: > > > >> I have setup pmacct with your help, and it's been running like a champ. > I > >> have also installed darkstat for comparison. I am seeing a big error > >>(around > >> 30%) between the 2 tools! > > ... > >> Here's what I am seeing: > >> > >> IP STARTENDDELTA > >> DARKSTAT(bytes) > >> 81.10.100.42 7607.7053 9477.4200 1869.7147 1,397,584,555 > >> 81.10.100.73 3603.2834 4716.6248 1113.3414 810,169,491 > >> 81.10.100.37 3540.3343 5698.6758 2158.3415 1,573,900,631 > >> 81.10.100.199 3444.3568 4358.3895914.0327 575,124,842 > >> 81.10.100.75 2951.8349 3697.5900 745.7551 556,560,149 > >> 81.10.100.30 2770.9552 3807.6038 1036.6486 715,830,077 > >> 81.10.100.46 2698.5764 3987.1379 1288.5615 856,582,079 > >> 81.10.100.44 1982.1858 2381.7297 399.5439 296,992,631 > >> 81.10.100.71 1880.2033 2522.7183 642.5150 548,180,038 > >> 81.10.100.201 1300.2739 2040.0713739.7974 411,031,858 > >> > >> Those are the top 10 BW users. All measurements are in MB (from SQL > >>query), > >> darkstat data is in bytes. As you can see, the first line it's 1.9GB vs > >> 1.4GB and so on ... > >> > >> Any ideas how to track such errors ? > > > > My first suspicion would be that Darkstat is reporting bytes transferred > > (TCP data) rather than total size of packets. You can confirm this with > > some simple tests. E.g. create a file of exactly 1MB on a remote web > > server and download it through your pmacct/darkstat box. If darkstat > > reports that the amount downloaded is just over 1MB (e.g. 1.001 MB) then > > it's reporting TCP data. > > > > pmacct will always report packet sizes (IP data) and therefore is likely > > to report more bytes downloaded. Given that the TCP overhead is about 40 > > bytes per 1500 byte packet, i.e. about 2.6%, I'd expect it to report > about > > 1.027 MB in this case. > > > > The overhead will be much higher for smaller packets which may explain > > your observed 30% discrepancy. If so, this is arguably a bug (or > > limitation) of darkstat rather than pmacct. > > > > Please let us know what you discover. > > > > Cheers, Chris. > > > > -- > Недвижимость - по-прежнему самое перспективное вложение денег! > http://www.tmk.by Тел. (029) 628 28 28, (017) 507 61 37 > > > > ___ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists > ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Measurement accuracy issues
Hello Ahmed, I only want to add that you must be strong ensure that you compare same flows of data. It is seem obviously but you can easy miss something in setting of both programs and compare hasn't any sense. Alex > Hi Ahmed, > > On Tue, 10 Jun 2008, Ahmed Kamal wrote: > >> I have setup pmacct with your help, and it's been running like a champ. I >> have also installed darkstat for comparison. I am seeing a big error >>(around >> 30%) between the 2 tools! > ... >> Here's what I am seeing: >> >> IP STARTENDDELTA >> DARKSTAT(bytes) >> 81.10.100.42 7607.7053 9477.4200 1869.7147 1,397,584,555 >> 81.10.100.73 3603.2834 4716.6248 1113.3414 810,169,491 >> 81.10.100.37 3540.3343 5698.6758 2158.3415 1,573,900,631 >> 81.10.100.199 3444.3568 4358.3895914.0327 575,124,842 >> 81.10.100.75 2951.8349 3697.5900 745.7551 556,560,149 >> 81.10.100.30 2770.9552 3807.6038 1036.6486 715,830,077 >> 81.10.100.46 2698.5764 3987.1379 1288.5615 856,582,079 >> 81.10.100.44 1982.1858 2381.7297 399.5439 296,992,631 >> 81.10.100.71 1880.2033 2522.7183 642.5150 548,180,038 >> 81.10.100.201 1300.2739 2040.0713739.7974 411,031,858 >> >> Those are the top 10 BW users. All measurements are in MB (from SQL >>query), >> darkstat data is in bytes. As you can see, the first line it's 1.9GB vs >> 1.4GB and so on ... >> >> Any ideas how to track such errors ? > > My first suspicion would be that Darkstat is reporting bytes transferred > (TCP data) rather than total size of packets. You can confirm this with > some simple tests. E.g. create a file of exactly 1MB on a remote web > server and download it through your pmacct/darkstat box. If darkstat > reports that the amount downloaded is just over 1MB (e.g. 1.001 MB) then > it's reporting TCP data. > > pmacct will always report packet sizes (IP data) and therefore is likely > to report more bytes downloaded. Given that the TCP overhead is about 40 > bytes per 1500 byte packet, i.e. about 2.6%, I'd expect it to report about > 1.027 MB in this case. > > The overhead will be much higher for smaller packets which may explain > your observed 30% discrepancy. If so, this is arguably a bug (or > limitation) of darkstat rather than pmacct. > > Please let us know what you discover. > > Cheers, Chris. -- Недвижимость - по-прежнему самое перспективное вложение денег! http://www.tmk.by Тел. (029) 628 28 28, (017) 507 61 37 ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Measurement accuracy issues
Hi Ahmed, On Tue, 10 Jun 2008, Ahmed Kamal wrote: > I have setup pmacct with your help, and it's been running like a champ. I > have also installed darkstat for comparison. I am seeing a big error (around > 30%) between the 2 tools! ... > Here's what I am seeing: > > IP STARTENDDELTA > DARKSTAT(bytes) > 81.10.100.42 7607.7053 9477.4200 1869.7147 1,397,584,555 > 81.10.100.73 3603.2834 4716.6248 1113.3414 810,169,491 > 81.10.100.37 3540.3343 5698.6758 2158.3415 1,573,900,631 > 81.10.100.199 3444.3568 4358.3895914.0327 575,124,842 > 81.10.100.75 2951.8349 3697.5900 745.7551 556,560,149 > 81.10.100.30 2770.9552 3807.6038 1036.6486 715,830,077 > 81.10.100.46 2698.5764 3987.1379 1288.5615 856,582,079 > 81.10.100.44 1982.1858 2381.7297 399.5439 296,992,631 > 81.10.100.71 1880.2033 2522.7183 642.5150 548,180,038 > 81.10.100.201 1300.2739 2040.0713739.7974 411,031,858 > > Those are the top 10 BW users. All measurements are in MB (from SQL query), > darkstat data is in bytes. As you can see, the first line it's 1.9GB vs > 1.4GB and so on ... > > Any ideas how to track such errors ? My first suspicion would be that Darkstat is reporting bytes transferred (TCP data) rather than total size of packets. You can confirm this with some simple tests. E.g. create a file of exactly 1MB on a remote web server and download it through your pmacct/darkstat box. If darkstat reports that the amount downloaded is just over 1MB (e.g. 1.001 MB) then it's reporting TCP data. pmacct will always report packet sizes (IP data) and therefore is likely to report more bytes downloaded. Given that the TCP overhead is about 40 bytes per 1500 byte packet, i.e. about 2.6%, I'd expect it to report about 1.027 MB in this case. The overhead will be much higher for smaller packets which may explain your observed 30% discrepancy. If so, this is arguably a bug (or limitation) of darkstat rather than pmacct. Please let us know what you discover. Cheers, Chris. -- Aptivate | http://www.aptivate.org | Phone: +44 1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES Aptivate is a not-for-profit company registered in England and Wales with company number 04980791. ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
