Re: [squid-users] Data usage reported in log files

2017-03-11 Thread Eliezer Croitoru
Hey Yosi,

Can you see if the differences is on the incoming or outgoing traffic?
Squid will only account for incoming and if you are using some kind of caching 
with the quick_abort and other partial content prefetch it would make sense 
that the actual consumption of the bits from the Internet to squid will not 
match from squid to clients.

If you can send me or share with others your squid.conf we might be able to 
understand if something there might cause such an issue.

Thanks,
Eliezer

* Feel free to contact me directly by skype or phone

Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: elie...@ngtech.co.il


-Original Message-
From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf 
Of Yosi Greenfield
Sent: Friday, March 10, 2017 11:47 PM
To: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] Data usage reported in log files

Gentlemen,

Thanks Antony. Yes, we are accounting for everything else. I'm talking about 
port 3128 and 3129 only. 

Any other traffic is being tracked both by netflow and tcpdump and they match. 
What does not match is 3128/9 and squid log.

I'll report back after the weekend if the discrepancy is all sslbump traffic.

Thank you all,
Yosi


-Original Message-
From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf 
Of Antony Stone
Sent: Friday, March 10, 2017 4:31 PM
To: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] Data usage reported in log files

On Friday 10 March 2017 at 22:22:59, Yuri Voinov wrote:

> Of course, there is no stream video from security cams, no voice IP, 
> no SIP, no torrents, no RDP, no other protocol. They simple does not 
> exists and we're all believe that's all not above over 1% of overall
traffic.
> Yes. Sure. Really.
> 
> Only web-surfing :) Sure :)

Thanks for the standard sarcasm.

Has it occurred to you that Yosi might have been measuring traffic to & from 
the IP of the Squid server, so as to ignore everything else he knows is 
happening on his network, so he can compare like with like?

My "not more than 1%" was for the additional traffic to/from the Squid server, 
other than HTTP/S.


Antony.

> 11.03.2017 3:19, Yuri Voinov пишет:
> > 11.03.2017 2:57, Antony Stone пишет:
> >> On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:
> >>> Gentlemen, and it never occurred to you that there are other types 
> >>> of traffic besides HTTP / HTTPS, right?
> >>> 
> >>> DNS, ICMP, other protocols?
> >> 
> >> I'm assuming Yosi has been measuring only TCP traffic, but even if 
> >> he's been measuring everything, I don't think DNS, ICMP and other 
> >> protocols would add more than 1% on top of HTTP/S, unless (as 
> >> Marcus suggested) there is also totally-non-Squid traffic on the link 
> >> being measured.
> > 
> > Come on, sure? Even in L7? Really? Cool story, bro!
> > 
> >> Antony.
> >> 
> >>> 11.03.2017 2:44, Yosi Greenfield пишет:
> >>>> Aha! That could be it. I use sslbump, but not for all users. I'll 
> >>>> check that out, although I think that it's a problem even for 
> >>>> bumped users. Even for bumped users we don't bump all sites, so 
> >>>> that really could be it.
> >>>> 
> >>>> Thanks!
> >>>> 
> >>>> 
> >>>> -Original Message-
> >>>> From: squid-users 
> >>>> [mailto:squid-users-boun...@lists.squid-cache.org]
> >>>> On Behalf Of Marcus Kool
> >>>> Sent: Friday, March 10, 2017 3:38 PM
> >>>> To: squid-users@lists.squid-cache.org
> >>>> Subject: Re: [squid-users] Data usage reported in log files
> >>>> 
> >>>> On 10/03/17 16:27, Yosi Greenfield wrote:
> >>>>> Thanks!
> >>>>> 
> >>>>> Netflow is much larger.
> >>>>> 
> >>>>> I really want to know exactly what site is costing my users data.
> >>>>> Many of our users are on metered connections and are paying for 
> >>>>> overage, but I can't tell where that overage is being used. Are 
> >>>>> they using youtube, webmail, wetransfer? I see only a fraction 
> >>>>> of their actual proxy usage in my squid logs.
> >>>>> 
> >>>>> Data compression would give the opposite result, so that's not 
> >>>>> what I'm seeing.
> >>>>> 
> >>>>> Any other ideas?
> >>>> 
> >>>> Is there any traf

Re: [squid-users] Data usage reported in log files

2017-03-10 Thread Yuri Voinov


11.03.2017 3:47, Yosi Greenfield пишет:
> Gentlemen,
>
> Thanks Antony. Yes, we are accounting for everything else. I'm
> talking about port 3128 and 3129 only. 
>
> Any other traffic is being tracked both by netflow and tcpdump and
> they match. What does not match is 3128/9 and squid log.
It can be also because of tunneled traffic.
>
> I'll report back after the weekend if the discrepancy is all
> sslbump traffic.
>
> Thank you all,
> Yosi
>
>
> -Original Message-
> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On
> Behalf Of Antony Stone
> Sent: Friday, March 10, 2017 4:31 PM
> To: squid-users@lists.squid-cache.org
> Subject: Re: [squid-users] Data usage reported in log files
>
> On Friday 10 March 2017 at 22:22:59, Yuri Voinov wrote:
>
>> Of course, there is no stream video from security cams, no voice IP, 
>> no SIP, no torrents, no RDP, no other protocol. They simple does not 
>> exists and we're all believe that's all not above over 1% of overall
> traffic.
>> Yes. Sure. Really.
>>
>> Only web-surfing :) Sure :)
> Thanks for the standard sarcasm.
>
> Has it occurred to you that Yosi might have been measuring traffic to & from
> the IP of the Squid server, so as to ignore everything else he knows is
> happening on his network, so he can compare like with like?
>
> My "not more than 1%" was for the additional traffic to/from the Squid
> server, other than HTTP/S.
>
>
> Antony.
>
>> 11.03.2017 3:19, Yuri Voinov пишет:
>>> 11.03.2017 2:57, Antony Stone пишет:
>>>> On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:
>>>>> Gentlemen, and it never occurred to you that there are other types of
>>>>> traffic besides HTTP / HTTPS, right?
>>>>>
>>>>> DNS, ICMP, other protocols?
>>>> I'm assuming Yosi has been measuring only TCP traffic, but even if he's
>>>> been measuring everything, I don't think DNS, ICMP and other protocols
>>>> would add more than 1% on top of HTTP/S, unless (as Marcus suggested)
>>>> there is also totally-non-Squid traffic on the link being measured.
>>> Come on, sure? Even in L7? Really? Cool story, bro!
>>>
>>>> Antony.
>>>>
>>>>> 11.03.2017 2:44, Yosi Greenfield пишет:
>>>>>> Aha! That could be it. I use sslbump, but not for all users. I'll
>>>>>> check that out, although I think that it's a problem even for bumped
>>>>>> users. Even for bumped users we don't bump all sites, so that really
>>>>>> could be it.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>> -Original Message-
>>>>>> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org]
>>>>>> On Behalf Of Marcus Kool
>>>>>> Sent: Friday, March 10, 2017 3:38 PM
>>>>>> To: squid-users@lists.squid-cache.org
>>>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>>>
>>>>>> On 10/03/17 16:27, Yosi Greenfield wrote:
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Netflow is much larger.
>>>>>>>
>>>>>>> I really want to know exactly what site is costing my users data.
>>>>>>> Many of our users are on metered connections and are paying for
>>>>>>> overage, but I can't tell where that overage is being used. Are they
>>>>>>> using youtube, webmail, wetransfer? I see only a fraction of their
>>>>>>> actual proxy usage in my squid logs.
>>>>>>>
>>>>>>> Data compression would give the opposite result, so that's not what
>>>>>>> I'm seeing.
>>>>>>>
>>>>>>> Any other ideas?
>>>>>> Is there any traffic that is not directed to Squid?
>>>>>>
>>>>>> Do you use ssl-bump in bump mode ?
>>>>>> If not, Squid has no idea how many bytes go through the (HTTPS)
>>>>>> tunnels.
>>>>>>
>>>>>> Marcus
>>>>>>
>>>>>>> -Original Message-
>>>>>>> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org]
>>>>>>> On Behalf Of Antony Stone
>>>>>>> Sent: Friday, March 10, 2017 2:21 PM
>>&

Re: [squid-users] Data usage reported in log files

2017-03-10 Thread Yuri Voinov


11.03.2017 3:43, Antony Stone пишет:
> On Friday 10 March 2017 at 22:33:44, Yuri Voinov wrote:
>
>> We have not seen the network topology and the full configuration of
>> network devices - what are we arguing about and guessing about?
> Nobody is arguing, and we are guessing so that we might be helpful to Yosi 
> who 
> asked the question.
Guessing can be worse than a lack of response. As they take them away
from the true picture. Especially when you do not have any facts.
>
> Incidentally, please could you consider putting all of your comments (which 
> are unrelated to further replies from other people) into a single posting, 
> instead of sending, for example, four emails to the list, each replying only 
> to your own previous comment?
>
> That would make things far easier to follow in the conversation.
I'll think about it in the future. I usually do not get into the
discussion here, except for very rare cases.
>
>
> Thanks,
>
>
> Antony.
>

-- 
Bugs to the Future


0x613DEC46.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users


Re: [squid-users] Data usage reported in log files

2017-03-10 Thread Yosi Greenfield
Gentlemen,

Thanks Antony. Yes, we are accounting for everything else. I'm
talking about port 3128 and 3129 only. 

Any other traffic is being tracked both by netflow and tcpdump and
they match. What does not match is 3128/9 and squid log.

I'll report back after the weekend if the discrepancy is all
sslbump traffic.

Thank you all,
Yosi


-Original Message-
From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On
Behalf Of Antony Stone
Sent: Friday, March 10, 2017 4:31 PM
To: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] Data usage reported in log files

On Friday 10 March 2017 at 22:22:59, Yuri Voinov wrote:

> Of course, there is no stream video from security cams, no voice IP, 
> no SIP, no torrents, no RDP, no other protocol. They simple does not 
> exists and we're all believe that's all not above over 1% of overall
traffic.
> Yes. Sure. Really.
> 
> Only web-surfing :) Sure :)

Thanks for the standard sarcasm.

Has it occurred to you that Yosi might have been measuring traffic to & from
the IP of the Squid server, so as to ignore everything else he knows is
happening on his network, so he can compare like with like?

My "not more than 1%" was for the additional traffic to/from the Squid
server, other than HTTP/S.


Antony.

> 11.03.2017 3:19, Yuri Voinov пишет:
> > 11.03.2017 2:57, Antony Stone пишет:
> >> On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:
> >>> Gentlemen, and it never occurred to you that there are other types of
> >>> traffic besides HTTP / HTTPS, right?
> >>> 
> >>> DNS, ICMP, other protocols?
> >> 
> >> I'm assuming Yosi has been measuring only TCP traffic, but even if he's
> >> been measuring everything, I don't think DNS, ICMP and other protocols
> >> would add more than 1% on top of HTTP/S, unless (as Marcus suggested)
> >> there is also totally-non-Squid traffic on the link being measured.
> > 
> > Come on, sure? Even in L7? Really? Cool story, bro!
> > 
> >> Antony.
> >> 
> >>> 11.03.2017 2:44, Yosi Greenfield пишет:
> >>>> Aha! That could be it. I use sslbump, but not for all users. I'll
> >>>> check that out, although I think that it's a problem even for bumped
> >>>> users. Even for bumped users we don't bump all sites, so that really
> >>>> could be it.
> >>>> 
> >>>> Thanks!
> >>>> 
> >>>> 
> >>>> -Original Message-
> >>>> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org]
> >>>> On Behalf Of Marcus Kool
> >>>> Sent: Friday, March 10, 2017 3:38 PM
> >>>> To: squid-users@lists.squid-cache.org
> >>>> Subject: Re: [squid-users] Data usage reported in log files
> >>>> 
> >>>> On 10/03/17 16:27, Yosi Greenfield wrote:
> >>>>> Thanks!
> >>>>> 
> >>>>> Netflow is much larger.
> >>>>> 
> >>>>> I really want to know exactly what site is costing my users data.
> >>>>> Many of our users are on metered connections and are paying for
> >>>>> overage, but I can't tell where that overage is being used. Are they
> >>>>> using youtube, webmail, wetransfer? I see only a fraction of their
> >>>>> actual proxy usage in my squid logs.
> >>>>> 
> >>>>> Data compression would give the opposite result, so that's not what
> >>>>> I'm seeing.
> >>>>> 
> >>>>> Any other ideas?
> >>>> 
> >>>> Is there any traffic that is not directed to Squid?
> >>>> 
> >>>> Do you use ssl-bump in bump mode ?
> >>>> If not, Squid has no idea how many bytes go through the (HTTPS)
> >>>> tunnels.
> >>>> 
> >>>> Marcus
> >>>> 
> >>>>> -Original Message-
> >>>>> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org]
> >>>>> On Behalf Of Antony Stone
> >>>>> Sent: Friday, March 10, 2017 2:21 PM
> >>>>> To: squid-users@lists.squid-cache.org
> >>>>> Subject: Re: [squid-users] Data usage reported in log files
> >>>>> 
> >>>>> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
> >>>>>> Hello all,
> >>>>>> 
> >>>>>> I'm ana

Re: [squid-users] Data usage reported in log files

2017-03-10 Thread Antony Stone
On Friday 10 March 2017 at 22:33:44, Yuri Voinov wrote:

> We have not seen the network topology and the full configuration of
> network devices - what are we arguing about and guessing about?

Nobody is arguing, and we are guessing so that we might be helpful to Yosi who 
asked the question.

Incidentally, please could you consider putting all of your comments (which 
are unrelated to further replies from other people) into a single posting, 
instead of sending, for example, four emails to the list, each replying only 
to your own previous comment?

That would make things far easier to follow in the conversation.


Thanks,


Antony.

-- 
I thought of going into banking, until I lost interest.

   Please reply to the list;
 please *don't* CC me.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users


Re: [squid-users] Data usage reported in log files

2017-03-10 Thread Yuri Voinov
According to the above, NetFlow will always show much more traffic than
the SQUID. This is obvious and there is nothing to discuss here. If this
is not clear to someone, put a collector that collects statistics at the
data link level and compare the counters. I'm not just talking about
TCP, Alex. There is also the UDP. And there are a lot of protocols that
squid can not see, including for the simple reason that these packets
are never routed to a SQUID.

We have not seen the network topology and the full configuration of
network devices - what are we arguing about and guessing about?


11.03.2017 3:27, Yuri Voinov пишет:
> Think of one simple thing. Squid does not see and can not see protocols
> that do not support. What do you expect from it? Does it work on L1/L2?
> No? Then what is the discussion about?
>
>
> 11.03.2017 3:22, Yuri Voinov пишет:
>> Of course, there is no stream video from security cams, no voice IP, no
>> SIP, no torrents, no RDP, no other protocol. They simple does not exists
>> and we're all believe that's all not above over 1% of overall traffic.
>> Yes. Sure. Really.
>>
>> Only web-surfing :) Sure :)
>>
>>
>> 11.03.2017 3:19, Yuri Voinov пишет:
>>> 11.03.2017 2:57, Antony Stone пишет:
>>>> On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:
>>>>
>>>>> Gentlemen, and it never occurred to you that there are other types of
>>>>> traffic besides HTTP / HTTPS, right?
>>>>>
>>>>> DNS, ICMP, other protocols?
>>>> I'm assuming Yosi has been measuring only TCP traffic, but even if he's 
>>>> been 
>>>> measuring everything, I don't think DNS, ICMP and other protocols would 
>>>> add 
>>>> more than 1% on top of HTTP/S, unless (as Marcus suggested) there is also 
>>>> totally-non-Squid traffic on the link being measured.
>>> Come on, sure? Even in L7? Really? Cool story, bro!
>>>> Antony.
>>>>
>>>>> 11.03.2017 2:44, Yosi Greenfield пишет:
>>>>>> Aha! That could be it. I use sslbump, but not for all users. I'll
>>>>>> check that out, although I think that it's a problem even for bumped
>>>>>> users. Even for bumped users we don't bump all sites, so that really
>>>>>> could be it.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>> -Original Message-
>>>>>> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On
>>>>>> Behalf Of Marcus Kool
>>>>>> Sent: Friday, March 10, 2017 3:38 PM
>>>>>> To: squid-users@lists.squid-cache.org
>>>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>>>
>>>>>> On 10/03/17 16:27, Yosi Greenfield wrote:
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Netflow is much larger.
>>>>>>>
>>>>>>> I really want to know exactly what site is costing my users data. Many
>>>>>>> of our users are on metered connections and are paying for overage,
>>>>>>> but I can't tell where that overage is being used. Are they using
>>>>>>> youtube, webmail, wetransfer? I see only a fraction of their actual
>>>>>>> proxy usage in my squid logs.
>>>>>>>
>>>>>>> Data compression would give the opposite result, so that's not what
>>>>>>> I'm seeing.
>>>>>>>
>>>>>>> Any other ideas?
>>>>>> Is there any traffic that is not directed to Squid?
>>>>>>
>>>>>> Do you use ssl-bump in bump mode ?
>>>>>> If not, Squid has no idea how many bytes go through the (HTTPS) tunnels.
>>>>>>
>>>>>> Marcus
>>>>>>
>>>>>>> -Original Message-
>>>>>>> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org]
>>>>>>> On Behalf Of Antony Stone
>>>>>>> Sent: Friday, March 10, 2017 2:21 PM
>>>>>>> To: squid-users@lists.squid-cache.org
>>>>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>>>>
>>>>>>> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
>>>>>>>> Hello all,
>>>>>>>>
>>>>>>>> I

Re: [squid-users] Data usage reported in log files

2017-03-10 Thread Antony Stone
On Friday 10 March 2017 at 22:22:59, Yuri Voinov wrote:

> Of course, there is no stream video from security cams, no voice IP, no
> SIP, no torrents, no RDP, no other protocol. They simple does not exists
> and we're all believe that's all not above over 1% of overall traffic.
> Yes. Sure. Really.
> 
> Only web-surfing :) Sure :)

Thanks for the standard sarcasm.

Has it occurred to you that Yosi might have been measuring traffic to & from 
the 
IP of the Squid server, so as to ignore everything else he knows is happening 
on his network, so he can compare like with like?

My "not more than 1%" was for the additional traffic to/from the Squid server, 
other than HTTP/S.


Antony.

> 11.03.2017 3:19, Yuri Voinov пишет:
> > 11.03.2017 2:57, Antony Stone пишет:
> >> On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:
> >>> Gentlemen, and it never occurred to you that there are other types of
> >>> traffic besides HTTP / HTTPS, right?
> >>> 
> >>> DNS, ICMP, other protocols?
> >> 
> >> I'm assuming Yosi has been measuring only TCP traffic, but even if he's
> >> been measuring everything, I don't think DNS, ICMP and other protocols
> >> would add more than 1% on top of HTTP/S, unless (as Marcus suggested)
> >> there is also totally-non-Squid traffic on the link being measured.
> > 
> > Come on, sure? Even in L7? Really? Cool story, bro!
> > 
> >> Antony.
> >> 
> >>> 11.03.2017 2:44, Yosi Greenfield пишет:
> >>>> Aha! That could be it. I use sslbump, but not for all users. I'll
> >>>> check that out, although I think that it's a problem even for bumped
> >>>> users. Even for bumped users we don't bump all sites, so that really
> >>>> could be it.
> >>>> 
> >>>> Thanks!
> >>>> 
> >>>> 
> >>>> -Original Message-
> >>>> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org]
> >>>> On Behalf Of Marcus Kool
> >>>> Sent: Friday, March 10, 2017 3:38 PM
> >>>> To: squid-users@lists.squid-cache.org
> >>>> Subject: Re: [squid-users] Data usage reported in log files
> >>>> 
> >>>> On 10/03/17 16:27, Yosi Greenfield wrote:
> >>>>> Thanks!
> >>>>> 
> >>>>> Netflow is much larger.
> >>>>> 
> >>>>> I really want to know exactly what site is costing my users data.
> >>>>> Many of our users are on metered connections and are paying for
> >>>>> overage, but I can't tell where that overage is being used. Are they
> >>>>> using youtube, webmail, wetransfer? I see only a fraction of their
> >>>>> actual proxy usage in my squid logs.
> >>>>> 
> >>>>> Data compression would give the opposite result, so that's not what
> >>>>> I'm seeing.
> >>>>> 
> >>>>> Any other ideas?
> >>>> 
> >>>> Is there any traffic that is not directed to Squid?
> >>>> 
> >>>> Do you use ssl-bump in bump mode ?
> >>>> If not, Squid has no idea how many bytes go through the (HTTPS)
> >>>> tunnels.
> >>>> 
> >>>> Marcus
> >>>> 
> >>>>> -Original Message-
> >>>>> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org]
> >>>>> On Behalf Of Antony Stone
> >>>>> Sent: Friday, March 10, 2017 2:21 PM
> >>>>> To: squid-users@lists.squid-cache.org
> >>>>> Subject: Re: [squid-users] Data usage reported in log files
> >>>>> 
> >>>>> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
> >>>>>> Hello all,
> >>>>>> 
> >>>>>> I'm analyzing my squid logs with sarg, and I see that the number of
> >>>>>> bytes reported as used by any particular user are often nowhere near
> >>>>>> the bytes reported by netflow and tcpdump.
> >>>>> 
> >>>>> Which is larger?
> >>>>> 
> >>>>>> I'm trying to trace my users' data usage by site, but I'm unable to
> >>>>>> do so from the log files because of this.
> >>>>> 
> >>>>> Well, what is it you really want to know?
> >>>>> 
> >>>>> netflow / tcpdump will give you accurate numbers for the quantity of
> >>>>> data on your Internet link - I assume this is what you're most
> >>>>> interested in?
> >>>>> Squid will show you what quantity of data goes to/from the clients,
> >>>>> but is that really important?
> >>>>> 
> >>>>>> Can someone please explain to me what I might be missing? Why does
> >>>>>> squid log report one thing and netflow and tcpdump show something
> >>>>>> else?
> >>>>> 
> >>>>> Data compression?
> >>>>> 
> >>>>> HTTP responses are often gzipped, so if tcpdump is showing you
> >>>>> smaller numbers of bytes than Squid reports, that's what I'd look at
> >>>>> first.
> >>>>> 
> >>>>> 
> >>>>> Antony.

-- 
 yes, but this is #lbw, we don't do normal

   Please reply to the list;
 please *don't* CC me.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users


Re: [squid-users] Data usage reported in log files

2017-03-10 Thread Yuri Voinov
Think of one simple thing. Squid does not see and can not see protocols
that do not support. What do you expect from it? Does it work on L1/L2?
No? Then what is the discussion about?


11.03.2017 3:22, Yuri Voinov пишет:
> Of course, there is no stream video from security cams, no voice IP, no
> SIP, no torrents, no RDP, no other protocol. They simple does not exists
> and we're all believe that's all not above over 1% of overall traffic.
> Yes. Sure. Really.
>
> Only web-surfing :) Sure :)
>
>
> 11.03.2017 3:19, Yuri Voinov пишет:
>> 11.03.2017 2:57, Antony Stone пишет:
>>> On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:
>>>
>>>> Gentlemen, and it never occurred to you that there are other types of
>>>> traffic besides HTTP / HTTPS, right?
>>>>
>>>> DNS, ICMP, other protocols?
>>> I'm assuming Yosi has been measuring only TCP traffic, but even if he's 
>>> been 
>>> measuring everything, I don't think DNS, ICMP and other protocols would add 
>>> more than 1% on top of HTTP/S, unless (as Marcus suggested) there is also 
>>> totally-non-Squid traffic on the link being measured.
>> Come on, sure? Even in L7? Really? Cool story, bro!
>>> Antony.
>>>
>>>> 11.03.2017 2:44, Yosi Greenfield пишет:
>>>>> Aha! That could be it. I use sslbump, but not for all users. I'll
>>>>> check that out, although I think that it's a problem even for bumped
>>>>> users. Even for bumped users we don't bump all sites, so that really
>>>>> could be it.
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>> -Original Message-
>>>>> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On
>>>>> Behalf Of Marcus Kool
>>>>> Sent: Friday, March 10, 2017 3:38 PM
>>>>> To: squid-users@lists.squid-cache.org
>>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>>
>>>>> On 10/03/17 16:27, Yosi Greenfield wrote:
>>>>>> Thanks!
>>>>>>
>>>>>> Netflow is much larger.
>>>>>>
>>>>>> I really want to know exactly what site is costing my users data. Many
>>>>>> of our users are on metered connections and are paying for overage,
>>>>>> but I can't tell where that overage is being used. Are they using
>>>>>> youtube, webmail, wetransfer? I see only a fraction of their actual
>>>>>> proxy usage in my squid logs.
>>>>>>
>>>>>> Data compression would give the opposite result, so that's not what
>>>>>> I'm seeing.
>>>>>>
>>>>>> Any other ideas?
>>>>> Is there any traffic that is not directed to Squid?
>>>>>
>>>>> Do you use ssl-bump in bump mode ?
>>>>> If not, Squid has no idea how many bytes go through the (HTTPS) tunnels.
>>>>>
>>>>> Marcus
>>>>>
>>>>>> -Original Message-
>>>>>> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org]
>>>>>> On Behalf Of Antony Stone
>>>>>> Sent: Friday, March 10, 2017 2:21 PM
>>>>>> To: squid-users@lists.squid-cache.org
>>>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>>>
>>>>>> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
>>>>>>> Hello all,
>>>>>>>
>>>>>>> I'm analyzing my squid logs with sarg, and I see that the number of
>>>>>>> bytes reported as used by any particular user are often nowhere near
>>>>>>> the bytes reported by netflow and tcpdump.
>>>>>> Which is larger?
>>>>>>
>>>>>>> I'm trying to trace my users' data usage by site, but I'm unable to
>>>>>>> do so from the log files because of this.
>>>>>> Well, what is it you really want to know?
>>>>>>
>>>>>> netflow / tcpdump will give you accurate numbers for the quantity of
>>>>>> data on your Internet link - I assume this is what you're most
>>>>>> interested in?
>>>>>> Squid will show you what quantity of data goes to/from the clients,
>>>>>> but is that really important?
>>>>>>
>>>>>>> Can someone please explain to me what I might be missing? Why does
>>>>>>> squid log report one thing and netflow and tcpdump show something
>>>>>>> else?
>>>>>> Data compression?
>>>>>>
>>>>>> HTTP responses are often gzipped, so if tcpdump is showing you smaller
>>>>>> numbers of bytes than Squid reports, that's what I'd look at first.
>>>>>>
>>>>>>
>>>>>> Antony.

-- 
Bugs to the Future


0x613DEC46.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users


Re: [squid-users] Data usage reported in log files

2017-03-10 Thread Yuri Voinov
Of course, there is no stream video from security cams, no voice IP, no
SIP, no torrents, no RDP, no other protocol. They simple does not exists
and we're all believe that's all not above over 1% of overall traffic.
Yes. Sure. Really.

Only web-surfing :) Sure :)


11.03.2017 3:19, Yuri Voinov пишет:
>
> 11.03.2017 2:57, Antony Stone пишет:
>> On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:
>>
>>> Gentlemen, and it never occurred to you that there are other types of
>>> traffic besides HTTP / HTTPS, right?
>>>
>>> DNS, ICMP, other protocols?
>> I'm assuming Yosi has been measuring only TCP traffic, but even if he's been 
>> measuring everything, I don't think DNS, ICMP and other protocols would add 
>> more than 1% on top of HTTP/S, unless (as Marcus suggested) there is also 
>> totally-non-Squid traffic on the link being measured.
> Come on, sure? Even in L7? Really? Cool story, bro!
>>
>> Antony.
>>
>>> 11.03.2017 2:44, Yosi Greenfield пишет:
>>>> Aha! That could be it. I use sslbump, but not for all users. I'll
>>>> check that out, although I think that it's a problem even for bumped
>>>> users. Even for bumped users we don't bump all sites, so that really
>>>> could be it.
>>>>
>>>> Thanks!
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On
>>>> Behalf Of Marcus Kool
>>>> Sent: Friday, March 10, 2017 3:38 PM
>>>> To: squid-users@lists.squid-cache.org
>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>
>>>> On 10/03/17 16:27, Yosi Greenfield wrote:
>>>>> Thanks!
>>>>>
>>>>> Netflow is much larger.
>>>>>
>>>>> I really want to know exactly what site is costing my users data. Many
>>>>> of our users are on metered connections and are paying for overage,
>>>>> but I can't tell where that overage is being used. Are they using
>>>>> youtube, webmail, wetransfer? I see only a fraction of their actual
>>>>> proxy usage in my squid logs.
>>>>>
>>>>> Data compression would give the opposite result, so that's not what
>>>>> I'm seeing.
>>>>>
>>>>> Any other ideas?
>>>> Is there any traffic that is not directed to Squid?
>>>>
>>>> Do you use ssl-bump in bump mode ?
>>>> If not, Squid has no idea how many bytes go through the (HTTPS) tunnels.
>>>>
>>>> Marcus
>>>>
>>>>> -Original Message-
>>>>> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org]
>>>>> On Behalf Of Antony Stone
>>>>> Sent: Friday, March 10, 2017 2:21 PM
>>>>> To: squid-users@lists.squid-cache.org
>>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>>
>>>>> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
>>>>>> Hello all,
>>>>>>
>>>>>> I'm analyzing my squid logs with sarg, and I see that the number of
>>>>>> bytes reported as used by any particular user are often nowhere near
>>>>>> the bytes reported by netflow and tcpdump.
>>>>> Which is larger?
>>>>>
>>>>>> I'm trying to trace my users' data usage by site, but I'm unable to
>>>>>> do so from the log files because of this.
>>>>> Well, what is it you really want to know?
>>>>>
>>>>> netflow / tcpdump will give you accurate numbers for the quantity of
>>>>> data on your Internet link - I assume this is what you're most
>>>>> interested in?
>>>>> Squid will show you what quantity of data goes to/from the clients,
>>>>> but is that really important?
>>>>>
>>>>>> Can someone please explain to me what I might be missing? Why does
>>>>>> squid log report one thing and netflow and tcpdump show something
>>>>>> else?
>>>>> Data compression?
>>>>>
>>>>> HTTP responses are often gzipped, so if tcpdump is showing you smaller
>>>>> numbers of bytes than Squid reports, that's what I'd look at first.
>>>>>
>>>>>
>>>>> Antony.

-- 
Bugs to the Future


0x613DEC46.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users


Re: [squid-users] Data usage reported in log files

2017-03-10 Thread Yuri Voinov


11.03.2017 2:57, Antony Stone пишет:
> On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:
>
>> Gentlemen, and it never occurred to you that there are other types of
>> traffic besides HTTP / HTTPS, right?
>>
>> DNS, ICMP, other protocols?
> I'm assuming Yosi has been measuring only TCP traffic, but even if he's been 
> measuring everything, I don't think DNS, ICMP and other protocols would add 
> more than 1% on top of HTTP/S, unless (as Marcus suggested) there is also 
> totally-non-Squid traffic on the link being measured.
Come on, sure? Even in L7? Really? Cool story, bro!
>
>
> Antony.
>
>> 11.03.2017 2:44, Yosi Greenfield пишет:
>>> Aha! That could be it. I use sslbump, but not for all users. I'll
>>> check that out, although I think that it's a problem even for bumped
>>> users. Even for bumped users we don't bump all sites, so that really
>>> could be it.
>>>
>>> Thanks!
>>>
>>>
>>> -Original Message-
>>> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On
>>> Behalf Of Marcus Kool
>>> Sent: Friday, March 10, 2017 3:38 PM
>>> To: squid-users@lists.squid-cache.org
>>> Subject: Re: [squid-users] Data usage reported in log files
>>>
>>> On 10/03/17 16:27, Yosi Greenfield wrote:
>>>> Thanks!
>>>>
>>>> Netflow is much larger.
>>>>
>>>> I really want to know exactly what site is costing my users data. Many
>>>> of our users are on metered connections and are paying for overage,
>>>> but I can't tell where that overage is being used. Are they using
>>>> youtube, webmail, wetransfer? I see only a fraction of their actual
>>>> proxy usage in my squid logs.
>>>>
>>>> Data compression would give the opposite result, so that's not what
>>>> I'm seeing.
>>>>
>>>> Any other ideas?
>>> Is there any traffic that is not directed to Squid?
>>>
>>> Do you use ssl-bump in bump mode ?
>>> If not, Squid has no idea how many bytes go through the (HTTPS) tunnels.
>>>
>>> Marcus
>>>
>>>> -Original Message-
>>>> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org]
>>>> On Behalf Of Antony Stone
>>>> Sent: Friday, March 10, 2017 2:21 PM
>>>> To: squid-users@lists.squid-cache.org
>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>
>>>> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
>>>>> Hello all,
>>>>>
>>>>> I'm analyzing my squid logs with sarg, and I see that the number of
>>>>> bytes reported as used by any particular user are often nowhere near
>>>>> the bytes reported by netflow and tcpdump.
>>>> Which is larger?
>>>>
>>>>> I'm trying to trace my users' data usage by site, but I'm unable to
>>>>> do so from the log files because of this.
>>>> Well, what is it you really want to know?
>>>>
>>>> netflow / tcpdump will give you accurate numbers for the quantity of
>>>> data on your Internet link - I assume this is what you're most
>>>> interested in?
>>>> Squid will show you what quantity of data goes to/from the clients,
>>>> but is that really important?
>>>>
>>>>> Can someone please explain to me what I might be missing? Why does
>>>>> squid log report one thing and netflow and tcpdump show something
>>>>> else?
>>>> Data compression?
>>>>
>>>> HTTP responses are often gzipped, so if tcpdump is showing you smaller
>>>> numbers of bytes than Squid reports, that's what I'd look at first.
>>>>
>>>>
>>>> Antony.

-- 
Bugs to the Future


0x613DEC46.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users


Re: [squid-users] Data usage reported in log files

2017-03-10 Thread Antony Stone
On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:

> Gentlemen, and it never occurred to you that there are other types of
> traffic besides HTTP / HTTPS, right?
> 
> DNS, ICMP, other protocols?

I'm assuming Yosi has been measuring only TCP traffic, but even if he's been 
measuring everything, I don't think DNS, ICMP and other protocols would add 
more than 1% on top of HTTP/S, unless (as Marcus suggested) there is also 
totally-non-Squid traffic on the link being measured.


Antony.

> 11.03.2017 2:44, Yosi Greenfield пишет:
> > Aha! That could be it. I use sslbump, but not for all users. I'll
> > check that out, although I think that it's a problem even for bumped
> > users. Even for bumped users we don't bump all sites, so that really
> > could be it.
> > 
> > Thanks!
> > 
> > 
> > -Original Message-
> > From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On
> > Behalf Of Marcus Kool
> > Sent: Friday, March 10, 2017 3:38 PM
> > To: squid-users@lists.squid-cache.org
> > Subject: Re: [squid-users] Data usage reported in log files
> > 
> > On 10/03/17 16:27, Yosi Greenfield wrote:
> >> Thanks!
> >> 
> >> Netflow is much larger.
> >> 
> >> I really want to know exactly what site is costing my users data. Many
> >> of our users are on metered connections and are paying for overage,
> >> but I can't tell where that overage is being used. Are they using
> >> youtube, webmail, wetransfer? I see only a fraction of their actual
> >> proxy usage in my squid logs.
> >> 
> >> Data compression would give the opposite result, so that's not what
> >> I'm seeing.
> >> 
> >> Any other ideas?
> > 
> > Is there any traffic that is not directed to Squid?
> > 
> > Do you use ssl-bump in bump mode ?
> > If not, Squid has no idea how many bytes go through the (HTTPS) tunnels.
> > 
> > Marcus
> > 
> >> -Original Message-
> >> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org]
> >> On Behalf Of Antony Stone
> >> Sent: Friday, March 10, 2017 2:21 PM
> >> To: squid-users@lists.squid-cache.org
> >> Subject: Re: [squid-users] Data usage reported in log files
> >> 
> >> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
> >>> Hello all,
> >>> 
> >>> I'm analyzing my squid logs with sarg, and I see that the number of
> >>> bytes reported as used by any particular user are often nowhere near
> >>> the bytes reported by netflow and tcpdump.
> >> 
> >> Which is larger?
> >> 
> >>> I'm trying to trace my users' data usage by site, but I'm unable to
> >>> do so from the log files because of this.
> >> 
> >> Well, what is it you really want to know?
> >> 
> >> netflow / tcpdump will give you accurate numbers for the quantity of
> >> data on your Internet link - I assume this is what you're most
> >> interested in?
> > 
> >> Squid will show you what quantity of data goes to/from the clients,
> >> but is that really important?
> >> 
> >>> Can someone please explain to me what I might be missing? Why does
> >>> squid log report one thing and netflow and tcpdump show something
> >>> else?
> >> 
> >> Data compression?
> >> 
> >> HTTP responses are often gzipped, so if tcpdump is showing you smaller
> >> numbers of bytes than Squid reports, that's what I'd look at first.
> >> 
> >> 
> >> Antony.

-- 
Normal people think "If it ain't broke, don't fix it".
Engineers think "If it ain't broke, it doesn't have enough features yet".

   Please reply to the list;
 please *don't* CC me.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users


Re: [squid-users] Data usage reported in log files

2017-03-10 Thread Alex Rousskov
On 03/10/2017 01:37 PM, Marcus Kool wrote:
> Squid has no idea how many bytes go through the (HTTPS) tunnels.

Actually, Squid knows the number of raw (encrypted) TCP payload bytes
inside a tunnel and should log that.

Squid also knows and logs the number of HTTP (decrypted) bytes if the
SSL tunnel is bumped. In that case, the logged number is often smaller
but could also be larger than the corresponding TCP payload, depending
on whether SSL uses compression.

In any case, Squid numbers do not contain TCP/IP/Ethernet headers and
control messages. They may also lack HTTP chunked encoding overheads.
Failed Squid-to-server connections are not logged if they were
successfully retried.

There are also logging/accounting bugs because there is currently no
automated system to detect them. For a recent example, see our fix at
http://bazaar.launchpad.net/~squid/squid/trunk/revision/14838

If you use the latest release and see a disparity (between Squid-logged
numbers and other sources of information) that cannot be explained by
known factors, consider reporting it.


Thank you,

Alex.

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users


Re: [squid-users] Data usage reported in log files

2017-03-10 Thread Yuri Voinov
Gentlemen, and it never occurred to you that there are other types of
traffic besides HTTP / HTTPS, right?

DNS, ICMP, other protocols?


11.03.2017 2:44, Yosi Greenfield пишет:
> Aha! That could be it. I use sslbump, but not for all users. I'll
> check that out, although I think that it's a problem even for bumped
> users. Even for bumped users we don't bump all sites, so that really
> could be it.
>
> Thanks!
>
>
> -Original Message-
> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On
> Behalf Of Marcus Kool
> Sent: Friday, March 10, 2017 3:38 PM
> To: squid-users@lists.squid-cache.org
> Subject: Re: [squid-users] Data usage reported in log files
>
>
>
> On 10/03/17 16:27, Yosi Greenfield wrote:
>> Thanks!
>>
>> Netflow is much larger.
>>
>> I really want to know exactly what site is costing my users data. Many 
>> of our users are on metered connections and are paying for overage, 
>> but I can't tell where that overage is being used. Are they using 
>> youtube, webmail, wetransfer? I see only a fraction of their actual 
>> proxy usage in my squid logs.
>>
>> Data compression would give the opposite result, so that's not what 
>> I'm seeing.
>>
>> Any other ideas?
> Is there any traffic that is not directed to Squid?
>
> Do you use ssl-bump in bump mode ?
> If not, Squid has no idea how many bytes go through the (HTTPS) tunnels.
>
> Marcus
>
>
>> -Original Message-----
>> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] 
>> On Behalf Of Antony Stone
>> Sent: Friday, March 10, 2017 2:21 PM
>> To: squid-users@lists.squid-cache.org
>> Subject: Re: [squid-users] Data usage reported in log files
>>
>> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
>>
>>> Hello all,
>>>
>>> I'm analyzing my squid logs with sarg, and I see that the number of 
>>> bytes reported as used by any particular user are often nowhere near 
>>> the bytes reported by netflow and tcpdump.
>> Which is larger?
>>
>>> I'm trying to trace my users' data usage by site, but I'm unable to 
>>> do so from the log files because of this.
>> Well, what is it you really want to know?
>>
>> netflow / tcpdump will give you accurate numbers for the quantity of 
>> data on your Internet link - I assume this is what you're most interested
> in?
>> Squid will show you what quantity of data goes to/from the clients, 
>> but is that really important?
>>
>>> Can someone please explain to me what I might be missing? Why does 
>>> squid log report one thing and netflow and tcpdump show something 
>>> else?
>> Data compression?
>>
>> HTTP responses are often gzipped, so if tcpdump is showing you smaller 
>> numbers of bytes than Squid reports, that's what I'd look at first.
>>
>>
>> Antony.
>>
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users

-- 
Bugs to the Future


0x613DEC46.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users


Re: [squid-users] Data usage reported in log files

2017-03-10 Thread Yosi Greenfield
Aha! That could be it. I use sslbump, but not for all users. I'll
check that out, although I think that it's a problem even for bumped
users. Even for bumped users we don't bump all sites, so that really
could be it.

Thanks!


-Original Message-
From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On
Behalf Of Marcus Kool
Sent: Friday, March 10, 2017 3:38 PM
To: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] Data usage reported in log files



On 10/03/17 16:27, Yosi Greenfield wrote:
> Thanks!
>
> Netflow is much larger.
>
> I really want to know exactly what site is costing my users data. Many 
> of our users are on metered connections and are paying for overage, 
> but I can't tell where that overage is being used. Are they using 
> youtube, webmail, wetransfer? I see only a fraction of their actual 
> proxy usage in my squid logs.
>
> Data compression would give the opposite result, so that's not what 
> I'm seeing.
>
> Any other ideas?

Is there any traffic that is not directed to Squid?

Do you use ssl-bump in bump mode ?
If not, Squid has no idea how many bytes go through the (HTTPS) tunnels.

Marcus


> -Original Message-
> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] 
> On Behalf Of Antony Stone
> Sent: Friday, March 10, 2017 2:21 PM
> To: squid-users@lists.squid-cache.org
> Subject: Re: [squid-users] Data usage reported in log files
>
> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
>
>> Hello all,
>>
>> I'm analyzing my squid logs with sarg, and I see that the number of 
>> bytes reported as used by any particular user are often nowhere near 
>> the bytes reported by netflow and tcpdump.
>
> Which is larger?
>
>> I'm trying to trace my users' data usage by site, but I'm unable to 
>> do so from the log files because of this.
>
> Well, what is it you really want to know?
>
> netflow / tcpdump will give you accurate numbers for the quantity of 
> data on your Internet link - I assume this is what you're most interested
in?
>
> Squid will show you what quantity of data goes to/from the clients, 
> but is that really important?
>
>> Can someone please explain to me what I might be missing? Why does 
>> squid log report one thing and netflow and tcpdump show something 
>> else?
>
> Data compression?
>
> HTTP responses are often gzipped, so if tcpdump is showing you smaller 
> numbers of bytes than Squid reports, that's what I'd look at first.
>
>
> Antony.
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users


Re: [squid-users] Data usage reported in log files

2017-03-10 Thread Marcus Kool



On 10/03/17 16:27, Yosi Greenfield wrote:

Thanks!

Netflow is much larger.

I really want to know exactly what site is costing my users data. Many of
our users are on metered connections and are paying for overage, but I can't
tell where that overage is being used. Are they using youtube, webmail,
wetransfer? I see only a fraction of their actual proxy usage in my squid
logs.

Data compression would give the opposite result, so that's not what I'm
seeing.

Any other ideas?


Is there any traffic that is not directed to Squid?

Do you use ssl-bump in bump mode ?
If not, Squid has no idea how many bytes go through the (HTTPS) tunnels.

Marcus



-Original Message-
From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On
Behalf Of Antony Stone
Sent: Friday, March 10, 2017 2:21 PM
To: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] Data usage reported in log files

On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:


Hello all,

I'm analyzing my squid logs with sarg, and I see that the number of
bytes reported as used by any particular user are often nowhere near
the bytes reported by netflow and tcpdump.


Which is larger?


I'm trying to trace my users' data usage by site, but I'm unable to do
so from the log files because of this.


Well, what is it you really want to know?

netflow / tcpdump will give you accurate numbers for the quantity of data on
your Internet link - I assume this is what you're most interested in?

Squid will show you what quantity of data goes to/from the clients, but is
that really important?


Can someone please explain to me what I might be missing? Why does
squid log report one thing and netflow and tcpdump show something
else?


Data compression?

HTTP responses are often gzipped, so if tcpdump is showing you smaller
numbers of bytes than Squid reports, that's what I'd look at first.


Antony.


___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users


Re: [squid-users] Data usage reported in log files

2017-03-10 Thread Yosi Greenfield
Thanks!

Netflow is much larger.

I really want to know exactly what site is costing my users data. Many of
our users are on metered connections and are paying for overage, but I can't
tell where that overage is being used. Are they using youtube, webmail,
wetransfer? I see only a fraction of their actual proxy usage in my squid
logs.

Data compression would give the opposite result, so that's not what I'm
seeing.

Any other ideas?


-Original Message-
From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On
Behalf Of Antony Stone
Sent: Friday, March 10, 2017 2:21 PM
To: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] Data usage reported in log files

On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:

> Hello all,
> 
> I'm analyzing my squid logs with sarg, and I see that the number of 
> bytes reported as used by any particular user are often nowhere near 
> the bytes reported by netflow and tcpdump.

Which is larger?

> I'm trying to trace my users' data usage by site, but I'm unable to do 
> so from the log files because of this.

Well, what is it you really want to know?

netflow / tcpdump will give you accurate numbers for the quantity of data on
your Internet link - I assume this is what you're most interested in?

Squid will show you what quantity of data goes to/from the clients, but is
that really important?

> Can someone please explain to me what I might be missing? Why does 
> squid log report one thing and netflow and tcpdump show something 
> else?

Data compression?

HTTP responses are often gzipped, so if tcpdump is showing you smaller
numbers of bytes than Squid reports, that's what I'd look at first.


Antony.

-- 
This sentence contains exacly three erors.

   Please reply to the list;
 please *don't* CC
me.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users


Re: [squid-users] Data usage reported in log files

2017-03-10 Thread Antony Stone
On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:

> Hello all,
> 
> I'm analyzing my squid logs with sarg, and I see that the number of
> bytes reported as used by any particular user are often nowhere
> near the bytes reported by netflow and tcpdump.

Which is larger?

> I'm trying to trace my users' data usage by site, but I'm unable to
> do so from the log files because of this.

Well, what is it you really want to know?

netflow / tcpdump will give you accurate numbers for the quantity of data on 
your Internet link - I assume this is what you're most interested in?

Squid will show you what quantity of data goes to/from the clients, but is 
that really important?

> Can someone please explain to me what I might be missing? Why does
> squid log report one thing and netflow and tcpdump show something
> else?

Data compression?

HTTP responses are often gzipped, so if tcpdump is showing you smaller numbers 
of bytes than Squid reports, that's what I'd look at first.


Antony.

-- 
This sentence contains exacly three erors.

   Please reply to the list;
 please *don't* CC me.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users