[squid-users] Peer2Peer Url categorizing, black\white lists, can it work?

2016-07-25 Thread Eliezer Croitoru 
I have it on my plate for quite some time and I was wondering about the
options and interest in the subject.

Intro:
Currently most free blacklists are distributed in the old fashion way of a
tar or other file.
There are benefits to these but I have not seen an option to be able to
"help" each other.
For example many proxy servers "knows" about a domain that other do not.
So even if the site exists and know in one side of the planet it's not in
another.
If it could be categorized or white\black listed in one side of the planet
why we cannot help each other?
Many admins adds sites to their DB and list but not many share them
publically.

The idea:
As an example Google and Mozilla services advertise malware infected sites
using their browser.
Many filtering solutions uses their clients logs to inspect and enhance
their lists.
There are many distributed key+value DB systems such as etcd and many others
DHT based.
I believe that somehow a url categorizing and black\white lists can be
advertised in a similar way.
The only limit is the "bootstap" or the "routers" of such a network.
Since such a service should only apply to KEYS and values which today should
not exceed 1MB I believe it would be pretty simple to create networks based
on that.
Once a network category or scheme can be defined it would be pretty simple
to "match" or "connect" between the relevant nodes.

Currently I am looking at the different options for the backend DB,
permissions and hierarchy which should give an admin a nice start point.
Such "online network" can be up and running pretty fast and it can enhance
the regular categories and lists to be more up-to-date.
Else then the actual categorizing and listing I believe that it would be
possible to share and generate a list of public domains which are known
compared to the current state which many parts of the web is "unknown".

If you wish to participate in any of the above ideas please contact me here
or privately.

Eliezer


Eliezer Croitoru  
Linux System Administrator
Mobile: +972-5-28704261
Email: elie...@ngtech.co.il
 


<>___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] cachemgr.cgi on embedded system

2016-07-25 Thread Amos Jeffries 
On 26/07/2016 12:49 a.m., reinerotto wrote:
> On 25/07/2016 8:55 p.m., reinerotto wrote:
>>> * Squid has Basic authentication enabled. <
>> This is _not_ the case in my environment.
>> I had an _impression_ from the wiki, that basic_auth _might_ be used.
>> (And there was a note from Yuri, having a similar problem like me :-)
>> Pls, consider an explicit statement in the wiki.
> 
> Which wiki page are you looking at? 
> 
> http://wiki.squid-cache.org/Features/CacheManager#default
> 

Ah. Thats not mentioning auth because the *default* is that no password
/ authentication is required, but some "reports" that affect Squid
behaviour are disabled.

Your setup needs Basic auth because you explicitly configured
cachemgr_passwd to apply for _all_ reports. Which is covered in the
section a few below that one about the default config.

I've updated that sectino to mention how Basic is involved.

Cheers
Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] cachemgr.cgi on embedded system

2016-07-25 Thread reinerotto 
On 25/07/2016 8:55 p.m., reinerotto wrote:
>> * Squid has Basic authentication enabled. <
> This is _not_ the case in my environment.
> I had an _impression_ from the wiki, that basic_auth _might_ be used.
> (And there was a note from Yuri, having a similar problem like me :-)
> Pls, consider an explicit statement in the wiki.

Which wiki page are you looking at? 

http://wiki.squid-cache.org/Features/CacheManager#default


Thanx for clarification. 



--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/cachemgr-cgi-on-embedded-system-tp4678665p4678677.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] cache peer communication about HIT/MISS between squid and and non-squid peer

2016-07-25 Thread Omid Kosari 
Following config in squid does not log anything

logformat nfmark %ts.%03tu %6tr %>a %Ss/%03>Hs %nfmark %http://squid-web-proxy-cache.1019090.n4.nabble.com/cache-peer-communication-about-HIT-MISS-between-squid-and-and-non-squid-peer-tp4600931p4678676.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.

2016-07-25 Thread Eliezer Croitoru 
Hey Omid,

I will comment inline.
And there are couple details which we need to understand couple issues.


Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: elie...@ngtech.co.il


-Original Message-
From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf 
Of Omid Kosari
Sent: Monday, July 25, 2016 12:15 PM
To: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] Windows Updates a Caching Stub zone, A windows 
updates store.

Hi,

Thanks for support .

recently i have seen a problem with version beta 0.2 . when fetcher is working 
the kernel logs lots of following error
TCP: out of memory -- consider tuning tcp_mem

# To verify the actual status we need the output of:
$ free -m
$ cat /proc/sys/net/ipv4/tcp_mem
$ top -n1 -b
$ cat /proc/net/sockstat
$ cat /proc/sys/net/ipv4/tcp_max_orphans 

I think the problem is about orphaned connections which i mentioned before .
Managed to try new version to see what happens.

# If you have an orphaned connections on the machine with or without the MS 
updates proxy, you should consider to analyze the machine structure and load in 
general.
If indeed there are orphan connections we need to verify if it's from the squid 
or my service or the combination of them together.


Also i have a feature request . Please provide a configuration file for example 
in /etc/foldername or even beside the binary files to have selective options 
for both fetcher and logger.

# With what options for the logger and fetcher?

I have seen following change log
beta 0.3 - 19/07/2016
+ Upgraded the fetcher to honour private and no-store cache-control  headers
when fetching objects.

As my point of view the more hits is better and there is no problem to store 
private and no-store objects if it helps to achieve more hits and bandwidth 
saving . So it would be fine to have an option in mentioned config file to 
change it myself .

# I understand your way of looking at things but this is a very wrong way to 
look at cache and store.
The problem with storing private and no-store responses is very simple.
These files are temporary and exists for one request only(in most cases).
Specifically for MS it is true and they do not use private files more then once.
I do not wish to offend you or anyone by not honoring such a request but since 
it's a public service this is the definition of it.
If you want to see the options of the fetcher and the service just add the "-h" 
option to see the available options.

I have considered to use some log file but yet to get to the point which I have 
a specific format that I want to work with.
I will try to see what can be done with log files and also what should be done 
to handle log rotation. 

Thanks again


## Resources
* http://blog.tsunanet.net/2011/03/out-of-socket-memory.html

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] What do the bytes and duration fields in squid log count for https (CONNECT)?

2016-07-25 Thread Amos Jeffries 
On 26/07/2016 12:04 a.m., Henry S. Thompson wrote:
> Amos Jeffries writes:
> 
>> On 25/07/2016 10:34 p.m., Henry S. Thompson wrote:
>>> Standard squid config only logs one CONNECT line for any https
>>> transaction. What is being counted/timed by the reported bytes and
>>> duration fields in that line?
>>>
>>> I'm guessing it's the total time taken and total bytes delivered to the
>>> client by any and all transactions in the course of the TLS connection
>>> established by that CONNECT, but I can't find anything in the log
>>> documentation which confirms that.
>>
>> Yes. There is no HTTPS or TLS as far as Squid is concerned. (In modern
>> traffic you are also very likely to be wrong about it being HTTPS or TLS
>> on port 443. The (browser?) URL saying "https://"; does not make it HTTPS
>> inside the tunnel).
> 
> Indeed, understood
> 
>> An HTTP CONNECT message with opaque data is all Squid sees. Its duration
>> is how long it takes, and the opaque data is the size it is.
> 
> Thanks for your reply, but this part leaves me confused.  The CONNECT
> message itself is short, as is the likely reply, and presumably doesn't
> take long to process.  But the times and sizes I'm seeing are long/big,
> so it doesn't seem likely that they are the time and size of the
> response to the CONNECT as such, which is what you appear to be saying
> above...
> 
> That is, what is the 'it' you refer to in your final sentence?

Sorry, coudl have been clearer.

Unless you are using SSL-Bump or such to process the contents specially.
The duration is from the CONNECT message arriving to the time TCP close
is used to end the tunnel. The size should be the bytes sent to the
client (excluding the 200 reply message itself) during that time.

Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] What do the bytes and duration fields in squid log count for https (CONNECT)?

2016-07-25 Thread Henry S. Thompson 
Amos Jeffries writes:

> On 25/07/2016 10:34 p.m., Henry S. Thompson wrote:
>> Standard squid config only logs one CONNECT line for any https
>> transaction. What is being counted/timed by the reported bytes and
>> duration fields in that line?
>> 
>> I'm guessing it's the total time taken and total bytes delivered to the
>> client by any and all transactions in the course of the TLS connection
>> established by that CONNECT, but I can't find anything in the log
>> documentation which confirms that.
>
> Yes. There is no HTTPS or TLS as far as Squid is concerned. (In modern
> traffic you are also very likely to be wrong about it being HTTPS or TLS
> on port 443. The (browser?) URL saying "https://"; does not make it HTTPS
> inside the tunnel).

Indeed, understood

> An HTTP CONNECT message with opaque data is all Squid sees. Its duration
> is how long it takes, and the opaque data is the size it is.

Thanks for your reply, but this part leaves me confused.  The CONNECT
message itself is short, as is the likely reply, and presumably doesn't
take long to process.  But the times and sizes I'm seeing are long/big,
so it doesn't seem likely that they are the time and size of the
response to the CONNECT as such, which is what you appear to be saying
above...

That is, what is the 'it' you refer to in your final sentence?

ht
-- 
   Henry S. Thompson, School of Informatics, University of Edinburgh
  10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
Fax: (44) 131 650-4587, e-mail: h...@inf.ed.ac.uk
   URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this -- mail without it is forged spam]
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] What do the bytes and duration fields in squid log count for https (CONNECT)?

2016-07-25 Thread Amos Jeffries 
On 25/07/2016 10:34 p.m., Henry S. Thompson wrote:
> Standard squid config only logs one CONNECT line for any https
> transaction. What is being counted/timed by the reported bytes and
> duration fields in that line?
> 
> I'm guessing it's the total time taken and total bytes delivered to the
> client by any and all transactions in the course of the TLS connection
> established by that CONNECT, but I can't find anything in the log
> documentation which confirms that.

Yes. There is no HTTPS or TLS as far as Squid is concerned. (In modern
traffic you are also very likely to be wrong about it being HTTPS or TLS
on port 443. The (browser?) URL saying "https://"; does not make it HTTPS
inside the tunnel).

An HTTP CONNECT message with opaque data is all Squid sees. Its duration
is how long it takes, and the opaque data is the size it is.

Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] cachemgr.cgi on embedded system

2016-07-25 Thread Amos Jeffries 
On 25/07/2016 8:55 p.m., reinerotto wrote:
>> * Squid has Basic authentication enabled. <
> This is _not_ the case in my environment. 
> I had an _impression_ from the wiki, that basic_auth _might_ be used.
> (And there was a note from Yuri, having a similar problem like me :-)
> Pls, consider an explicit statement in the wiki.

Which wiki page are you looking at?

>  
> On an embedded device, a  _default_ squid install (used 2.7 in the past,
> which was good enough for a long time) 
> eats up a lot of precious non-volatile memory (16MB flash mem are already
> "plenty") , so
> I have to scale down squid to required functionality only, when compiling.
> So, most likely these are my .config-options, relevant to my problem:
> 
> '--disable-external-acl-helpers' 
> '--disable-auth-negotiate' 
> '--disable-auth-ntlm' 
> '--disable-auth-digest' 
> '--disable-auth-basic'
> 
> I assume, '--enable-auth-basic' is required for cachemgr to work, 

Yes, thats the one. You will need it to be enabled, but can list "none"
to build no helpers.

> but what about '--disable-external-acl-helpers' ?
>  

Only if you use external ACL helpersthat are bundled with Squid. That
does not remove the external ACL code from Squid, just controlls the
bundled helpers being built (or not).

For minimal install options you may want to compare the list in
test-suite/buildtests/layer-01-minimal.opts to yours. It is overdue for
updating right now, but might give you some other ideas of possible
shrinkage.

Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] What do the bytes and duration fields in squid log count for https (CONNECT)?

2016-07-25 Thread Henry S. Thompson 
Standard squid config only logs one CONNECT line for any https
transaction. What is being counted/timed by the reported bytes and
duration fields in that line?

I'm guessing it's the total time taken and total bytes delivered to the
client by any and all transactions in the course of the TLS connection
established by that CONNECT, but I can't find anything in the log
documentation which confirms that.

Thanks,

ht
-- 
   Henry S. Thompson, School of Informatics, University of Edinburgh
  10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
Fax: (44) 131 650-4587, e-mail: h...@inf.ed.ac.uk
   URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this -- mail without it is forged spam]
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.

2016-07-25 Thread Omid Kosari 
Hi,

Thanks for support .

recently i have seen a problem with version beta 0.2 . when fetcher is
working the kernel logs lots of following error
TCP: out of memory -- consider tuning tcp_mem

I think the problem is about orphaned connections which i mentioned before .
Managed to try new version to see what happens.

Also i have a feature request . Please provide a configuration file for
example in /etc/foldername or even beside the binary files to have selective
options for both fetcher and logger . 

I have seen following change log
beta 0.3 - 19/07/2016
+ Upgraded the fetcher to honour private and no-store cache-control headers
when fetching objects.

As my point of view the more hits is better and there is no problem to store
private and no-store objects if it helps to achieve more hits and bandwidth
saving . So it would be fine to have an option in mentioned config file to
change it myself .

Thanks again



--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4678669.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] cachemgr.cgi on embedded system

2016-07-25 Thread reinerotto 
>* Squid has Basic authentication enabled. <
This is _not_ the case in my environment. 
I had an _impression_ from the wiki, that basic_auth _might_ be used.
(And there was a note from Yuri, having a similar problem like me :-)
Pls, consider an explicit statement in the wiki.
 
On an embedded device, a  _default_ squid install (used 2.7 in the past,
which was good enough for a long time) 
eats up a lot of precious non-volatile memory (16MB flash mem are already
"plenty") , so
I have to scale down squid to required functionality only, when compiling.
So, most likely these are my .config-options, relevant to my problem:

'--disable-external-acl-helpers' 
'--disable-auth-negotiate' 
'--disable-auth-ntlm' 
'--disable-auth-digest' 
'--disable-auth-basic'

I assume, '--enable-auth-basic' is required for cachemgr to work, 
but what about '--disable-external-acl-helpers' ?
 





--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/cachemgr-cgi-on-embedded-system-tp4678665p4678668.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] cachemgr.cgi on embedded system

2016-07-25 Thread Amos Jeffries 
On 25/07/2016 7:16 a.m., Eliezer Croitoru wrote:
> Hey,
> 
> What version are you using?
> Squid since version 3.X has a built in interface which might fit your needs.
> You can see an example of usage at:
> http://wiki.squid-cache.org/Features/CacheManager#default
> 
> What you will need to do is to access the proxy directly using a url like:
> http://mycache.example.com:3128/squid-internal-mgr/menu
> 
> and for the info page from the menu:
> http://mycache.example.com:3128/squid-internal-mgr/info
> 
> So unless you have a special need for the cache manger cgi you should use the 
> http one.
> 

NP: the cachemgr.cgi tool from recent Squid releases will test for and
use that http:// interface instead of the old cache_proto:// scheme.

> 
> -Original Message-
> From: reinerotto
> 
> I have a problem to use cachemgr.cgi on an embedded system: 
> (Cache Server: 127.0.0.1:3128; manager name: manager: Password: maypasswd)

 "maypasswd" or "mypasswd"? If that is not a typo in your email it will
be the problem.


> browser:
> The following error was encountered while trying to retrieve the URL:
> cache_object://127.0.0.1/
> Cache Manager Access Denied.
> Sorry, you are not currently allowed to request cache_object://127.0.0.1/ 
> from this cache manager until you have authenticated yourself.
> ACL Access Denied
> 
> cache.log:
> 2016/07/24 13:19:00| CacheManager: unknown@local=127.0.0.1:3128
> remote=127.0.0.1:56590 FD 18 flags=1: password needed for 'menu'
> 
> squid.conf:
> acl manager proto cache_object
> #next just for testing
> http_access allow manager all
> cachemgr_passwd mypasswd all

Order is important. Where you put these lines in relation to any other
http_access rules matters a lot.
The current release recommend placing the http_access rules for manager
below the default "deny CONNECT !SSL_port" rule, above any other custom
rules you have.

> 
> On the embedded system, there is only a small http-server (uhttpd) running, 
> _not_ apache or similar, so I suspect some special "requirement" not met on 
> my system.
> It could be _either_ some special .configure option for squid (I have a 
> downsized one, self-compiled) _or_ some speciality regarding my http-server, 
> which otherwise works well.
> 

That should be fine as long as:

* the uhttpd can pass Basic authentication headers and the user-info
field of URLs through to the CGI tool.

* Squid is a current/recent release of Squid *and* cachemgr.cgi tool.

* Squid has Basic authentication enabled.

Note that a current Squid supporting the new interface should be warning
you about incorrect manager ACL definition and refusing to startup using
the config mentioned above. "proto" is no longer the correct ACL type
for manager. There is a built-in one instead.


In your current system setup I suggest going with the default squid.conf
http_access manager lines. They are sufficient for a cachmgr.cgi tool
running on the same machine.


However, since cachemgr.cgi does not have to run on the embeded device
you can save a fair bit of space by placing it on an administrative web
server machine. For that you need to change the "http_access allow
manager localhost" to use an ACL checking for that machines IP instead
of localhost.


Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users