Squid Idea

[Thomas Grant (Messagelabs)]

Hi There,

 

Is it possible using Squid to say the following:

 

If browsing all sites, go to PROXY1.MYPROXY.COM however if browsing
BBC.CO.UK, GOOGLE.COM, HOTMAIL.CO.UK (examples only) go to
PROXY2.MYPROXY.COM?

 

Basically, the option to set a different web proxy address for a given
list of IP or Domains that you are trying to reach?

 

Thanks

Tom

 

Kind Regards

 

Tom Grant


Support Centre Analyst

Symantec Hosted Services
www.messagelabs.com    


 

24x7 Global Client Support Centre: 
US/Canada: +1 (866) 807 6047 
EMEA: +44 (0) 870 850 3014 
Australia: 1 (800) 088099 
Hong Kong: 1 (800) 901220 
Asia Pacific: +852 6902 1130
supp...@messagelabs.com   


 

 

 

MessageLabs Ltd, a company registered in England number 3834506, 1240
Lansdowne Court, Gloucester Business Park, Gloucester, GL3 4AB

 


__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
__<>

Re: Squid Idea

[Pieter De Wit]

Hi Thomas,

You sure can - the option is called "cache_peer_acl" - google it for some 
examples.

BTW - This is the squid-dev group, not the "general support" one - I think your 
query is more suited to the general support one. From the top of my head, it's 
email is sq...@squid-cache.org

Cheers,

Pieter
  - Original Message - 
  From: Thomas Grant (Messagelabs) 
  To: squid-dev@squid-cache.org 
  Sent: Wednesday, December 16, 2009 03:03
  Subject: Squid Idea


  Hi There,

   

  Is it possible using Squid to say the following:

   

  If browsing all sites, go to PROXY1.MYPROXY.COM however if browsing 
BBC.CO.UK, GOOGLE.COM, HOTMAIL.CO.UK (examples only) go to PROXY2.MYPROXY.COM?

   

  Basically, the option to set a different web proxy address for a given list 
of IP or Domains that you are trying to reach?

   

  Thanks

  Tom

   

  Kind Regards

   

  Tom Grant


  Support Centre Analyst

  Symantec Hosted Services
  www.messagelabs.com  
  

   

  24x7 Global Client Support Centre: 
  US/Canada: +1 (866) 807 6047 
  EMEA: +44 (0) 870 850 3014 
  Australia: 1 (800) 088099 
  Hong Kong: 1 (800) 901220 
  Asia Pacific: +852 6902 1130
  supp...@messagelabs.com 
  

   



   

  MessageLabs Ltd, a company registered in England number 3834506, 1240 
Lansdowne Court, Gloucester Business Park, Gloucester, GL3 4AB

   


  __
  This email has been scanned by the MessageLabs Email Security System.
  For more information please visit http://www.messagelabs.com/email 
  __
<>

[PATCH] Append _ABORTED or _TIMEDOUT suffixes to the squid result code field

[Tsantilas Christos]

Hi all,
this patch append the _ABORTED or _TIMEDOUT suffixes to the action 
access.log field.

The patch also exist for squid3.1 branch.

The development sponsored by the Measurement Factory


Description:
  When an HTTP connection with a client times out, append _TIMEDOUT 
suffix to the Squid result code field in access.log.


When an HTTP connection with the client is terminated prematurely by
Squid, append _ABORTED suffix to the result code field in access.log.
Premature connection termination may happen when, for example, I/O
errors or server side-problems force Squid to terminate the master
transaction and close all associated connections.

The above changes make it possible to identify failed transactions even
when they have 200/200 received/send response status codes and a
"successful" Squid result code (e.g., TCP_MISS). This is important when
one does not want 1-hour "stuck" transactions for 15-byte GIFs to
significantly skew the mean response time statistics. Such transactions
eventually terminate due to, say, TCP errors, and the old code would
record huge response times for successfully-looking transactions.

-
Christos
=== modified file 'src/AccessLogEntry.h'
--- src/AccessLogEntry.h	2009-08-23 09:30:49 +
+++ src/AccessLogEntry.h	2009-10-19 06:30:43 +
@@ -55,12 +55,20 @@
 {
 
 public:
-HttpDetails() : method(METHOD_NONE), code(0), content_type(NULL) {}
+HttpDetails() : method(METHOD_NONE), code(0), content_type(NULL),
+	timedout(false), aborted(false) {}
 
 HttpRequestMethod method;
 int code;
 const char *content_type;
 HttpVersion version;
+bool timedout; ///< terminated due to a lifetime or I/O timeout
+bool aborted; ///< other abnormal termination (e.g., I/O error)
+
+/// compute suffix for the status access.log field
+const char *statusSfx() const {
+return timedout ? "_TIMEDOUT" : (aborted ? "_ABORTED" : "");
+}
 } http;
 
 class ICPDetails

=== modified file 'src/client_side.cc'
--- src/client_side.cc	2009-11-18 12:37:56 +
+++ src/client_side.cc	2009-12-15 18:17:24 +
@@ -623,6 +623,15 @@
 }
 }
 
+/// propagates abort event to all contexts
+void
+ConnStateData::notifyAllContexts(int xerrno)
+{
+typedef ClientSocketContext::Pointer CSCP;
+for (CSCP c = getCurrentContext(); c.getRaw(); c = c->next)
+c->noteIoError(xerrno);
+}
+
 /* This is a handler normally called by comm_close() */
 void ConnStateData::connStateClosed(const CommCloseCbParams &io)
 {
@@ -1622,6 +1631,19 @@
 context->writeComplete (fd, bufnotused, size, errflag);
 }
 
+/// remembers the abnormal connection termination for logging purposes
+void
+ClientSocketContext::noteIoError(const int xerrno)
+{
+if (http) {
+if (xerrno == ETIMEDOUT)
+http->al.http.timedout = true;
+else // even if xerrno is zero (which means read abort/eof)
+http->al.http.aborted = true;
+}
+}
+
+
 void
 ClientSocketContext::doClose()
 {
@@ -2215,6 +2237,7 @@
 } else if (!Config.onoff.half_closed_clients) {
 /* admin doesn't want to support half-closed client sockets */
 debugs(33, 3, "connFinishedWithConn: FD " << fd << " aborted (half_closed_clients disabled)");
+notifyAllContexts(0); // no specific error implies abort
 return 1;
 }
 }
@@ -2663,6 +2686,7 @@
  * lame half-close detection
  */
 if (connReadWasError(io.flag, io.size, io.xerrno)) {
+notifyAllContexts(io.xerrno);
 comm_close(fd);
 return;
 }
@@ -2937,8 +2961,9 @@
 clientLifetimeTimeout(int fd, void *data)
 {
 ClientHttpRequest *http = (ClientHttpRequest *)data;
-debugs(33, 1, "WARNING: Closing client " << http->getConn()->peer << " connection due to lifetime timeout");
+debugs(33, 1, "WARNING: Closing client " << " connection due to lifetime timeout");
 debugs(33, 1, "\t" << http->uri);
+http->al.http.timedout = true;
 comm_close(fd);
 }
 

=== modified file 'src/client_side.h'
--- src/client_side.h	2009-10-31 10:06:17 +
+++ src/client_side.h	2009-12-15 18:17:24 +
@@ -114,6 +114,7 @@
 void deferRecipientForLater(clientStreamNode * node, HttpReply * rep, StoreIOBuffer receivedData);
 bool multipartRangeRequest() const;
 void registerWithConn();
+void noteIoError(const int xerrno); ///< update state to reflect I/O error
 
 private:
 CBDATA_CLASS(ClientSocketContext);
@@ -142,6 +143,7 @@
 int getAvailableBufferLength() const;
 bool areAllContextsForThisConnection() const;
 void freeAllContexts();
+void notifyAllContexts(const int xerrno); ///< tell everybody about the err
 void readNextRequest();
 void makeSpaceAvailable();
 ClientSocketContext::Pointer getCurrentContext() const;

=== modified file 'src/log/access_log.cc'
--- src/log/access_log.cc	2009-11-22 20:37:27 +
+++ src/log/access_log.cc	2009-12-1

[PATCH] add support for write I/O timeout

[Tsantilas Christos]

This patch add support for write timeouts.
The patch also exist for squid3.1 branch
The development sponsored by the Measurement Factory


Description:
The write I/O timeout should trigger if Squid has data to write but
the connection is not ready to accept more data for the specified time.
If the write times out, the Comm caller's write handler is called with
an ETIMEDOUT COMM_ERROR error.

Comm may process a single write request in several chunks, without
caller's knowledge. The waiting time is reset internally by Comm after
each chunk is written.

Default timeout value is 15 minutes.

The implementation requires no changes in Comm callers but it adds write
timeouts to all connections, including the connections that have
context-specific write timeouts. I think that is fine.


-
 Christos
=== modified file 'src/cf.data.pre'
--- src/cf.data.pre	2009-11-29 05:33:51 +
+++ src/cf.data.pre	2009-12-15 18:17:23 +
@@ -3934,6 +3934,21 @@
 	default is 15 minutes.
 DOC_END
 
+NAME: write_timeout
+COMMENT: time-units
+TYPE: time_t
+LOC: Config.Timeout.write
+DEFAULT: 15 minutes
+DOC_START
+	This timeout is tracked for all connections that have data
+	available for writing and are waiting for the socket to become
+	ready. After each successful write, the timeout is extended by
+	the configured amount. If Squid has data to write but the
+	connection is not ready for the configured duration, the
+	transaction associated with the connection is terminated. The
+	default is 15 minutes.
+DOC_END
+
 NAME: request_timeout
 TYPE: time_t
 LOC: Config.Timeout.request

=== modified file 'src/comm.cc'
--- src/comm.cc	2009-11-17 10:07:53 +
+++ src/comm.cc	2009-12-15 18:17:24 +
@@ -1991,6 +1991,9 @@
 debugs(5, 5, "commHandleWrite: write() returns " << len);
 fd_bytes(fd, len, FD_WRITE);
 statCounter.syscalls.sock.writes++;
+// After each successful partial write, 
+// reset fde::writeStart to the current time. 
+fd_table[fd].writeStart = squid_curtime;
 
 if (len == 0) {
 /* Note we even call write if nleft == 0 */
@@ -2062,6 +2065,7 @@
 comm_io_callback_t *ccb = COMMIO_FD_WRITECB(fd);
 assert(!ccb->active());
 
+fd_table[fd].writeStart = squid_curtime;
 /* Queue the write */
 commio_set_callback(fd, IOCB_WRITE, ccb, callback,
 (char *)buf, free_func, size);
@@ -2162,6 +2166,18 @@
 return false;
 }
 
+static bool
+writeTimedOut(int fd)
+{
+if (!commio_has_callback(fd, IOCB_WRITE, COMMIO_FD_WRITECB(fd)))
+return false;
+
+if ((squid_curtime - fd_table[fd].writeStart) < Config.Timeout.write)
+return false;
+
+return true;
+}
+
 void
 checkTimeouts(void)
 {
@@ -2172,7 +2188,11 @@
 for (fd = 0; fd <= Biggest_FD; fd++) {
 F = &fd_table[fd];
 
-if (AlreadyTimedOut(F))
+if (writeTimedOut(fd)) {
+// We have an active write callback and we are timed out
+commio_finish_callback(fd, COMMIO_FD_WRITECB(fd), COMM_ERROR, ETIMEDOUT);
+}
+else if (AlreadyTimedOut(F))
 continue;
 
 debugs(5, 5, "checkTimeouts: FD " << fd << " Expired");

=== modified file 'src/fde.h'
--- src/fde.h	2009-01-07 11:24:40 +
+++ src/fde.h	2009-10-19 06:32:31 +
@@ -94,6 +94,7 @@
 void *write_data;
 AsyncCall::Pointer timeoutHandler;
 time_t timeout;
+time_t writeStart;
 void *lifetime_data;
 AsyncCall::Pointer closeHandler;
 AsyncCall::Pointer halfClosedReader; /// read handler for half-closed fds

=== modified file 'src/structs.h'
--- src/structs.h	2009-11-22 20:37:27 +
+++ src/structs.h	2009-12-15 18:17:24 +
@@ -163,6 +163,7 @@
 
 struct {
 time_t read;
+time_t write;
 time_t lifetime;
 time_t connect;
 time_t forward;

Re: Squid Idea

[Amos Jeffries]

On Wed, 16 Dec 2009 05:56:09 +1300, "Pieter De Wit" 
wrote:
> Hi Thomas,
> 
> You sure can - the option is called "cache_peer_acl" - google it for
some
> examples.
> 

cache_peer_access

> BTW - This is the squid-dev group, not the "general support" one - I
think
> your query is more suited to the general support one. From the top of my
> head, it's email is sq...@squid-cache.org
> 

Mailing list: squid-us...@squid-cache.org

Cheers
Amos

Re: [PATCH] Append _ABORTED or _TIMEDOUT suffixes to the squid result code field

[Amos Jeffries]

On Tue, 15 Dec 2009 21:56:08 +0200, Tsantilas Christos
 wrote:
> Hi all,
>  this patch append the _ABORTED or _TIMEDOUT suffixes to the action 
> access.log field.
> The patch also exist for squid3.1 branch.
> 
> The development sponsored by the Measurement Factory
> 
> 
> Description:
>When an HTTP connection with a client times out, append _TIMEDOUT 
> suffix to the Squid result code field in access.log.
> 
> When an HTTP connection with the client is terminated prematurely by
> Squid, append _ABORTED suffix to the result code field in access.log.
> Premature connection termination may happen when, for example, I/O
> errors or server side-problems force Squid to terminate the master
> transaction and close all associated connections.
> 
> The above changes make it possible to identify failed transactions even
> when they have 200/200 received/send response status codes and a
> "successful" Squid result code (e.g., TCP_MISS). This is important when
> one does not want 1-hour "stuck" transactions for 15-byte GIFs to
> significantly skew the mean response time statistics. Such transactions
> eventually terminate due to, say, TCP errors, and the old code would
> record huge response times for successfully-looking transactions.
> 
> -
> Christos

? This seems like a situation that should never occur...

Aborted and Timed Out transactions are 4xx/5xx errors from the client
viewpoint right?
or if they are really stale-on-error type pages we need to use MISS_STALE
tags.

Amos

Re: Assertion in clientProcessBody

[Mark Nottingham]

On 08/12/2009, at 4:12 PM, Henrik Nordstrom wrote:

> tis 2009-12-08 klockan 13:34 +1100 skrev Mark Nottingham:
> 
>> Any thoughts here? Should this really be >=, or should clientProcessBody 
>> never get a 0 size_left?
> 
> It's done when size_left == 0, and no further body processing handler
> shoud be active on this request at that time. Any data on the connection
> at this time is either surplus data (HTTP violation) or a pipelined
> request waiting to be processed.
> 
> If you look a little further down (about one screen) in
> clientProcessBody you'll also see that the body reader gets unregistered
> when processing reaches 0.

is that this?

/* Remove request link if this is the last part of the body, as
 * clientReadRequest automatically continues to process next request */
if (conn->body.size_left <= 0 && request != NULL)
requestUnregisterBody(request, clientReadBody, conn);

I think what's happening here is that conn->body.size_left is 0, but request 
*is* NULL, because it was a cache hit and clientWriteComplete decided to eat 
the request body. More verbose logging gave:

2009/12/15 17:09:15| clientReadBody: start fd=35 body_size=226 in.offset=211 
cb=0x4562cd req=0xc52f60
2009/12/15 17:09:15| clientProcessBody: start fd=35 body_size=226 in.offset=211 
cb=0x4562cd req=0xc52f60
2009/12/15 17:09:15| clientProcessBody: end fd=35 size=211 body_size=15 
in.offset=0 cb=0x4562cd req=0xc52f60
2009/12/15 17:09:15| clientReadRequest: FD 35: reading request...
2009/12/15 17:09:15| clientReadRequest: FD 35: read 15 bytes
2009/12/15 17:09:15| clientWriteComplete: FD 35, sz 3705, err 0, off 4200, len 
4200
2009/12/15 17:09:15| clientWriteComplete: FD 35 transfer is DONE
2009/12/15 17:09:15| clientWriteComplete: closing, but first we need to read 
the rest of the request
2009/12/15 17:09:15| clientProcessBody: start fd=35 body_size=15 in.offset=15 
cb=0x42ed4c req=(nil)
2009/12/15 17:09:15| clientEatRequestBodyHandler: FD 35 Keeping Alive
2009/12/15 17:09:15| clientKeepaliveNextRequest: FD 35
2009/12/15 17:09:15| httpRequestFree: [url]
2009/12/15 17:09:15| clientKeepaliveNextRequest: FD 35 reading next req
2009/12/15 17:09:15| clientReadRequest: FD 35: reading request...
2009/12/15 17:09:15| clientReadRequest: FD 35: read -1 bytes
2009/12/15 17:09:15| clientProcessBody: end fd=35 size=15 body_size=0 
in.offset=0 cb=0x42ed4c req=(nil)
2009/12/15 17:09:15| clientReadBody: start fd=35 body_size=0 in.offset=0 
cb=0x4562cd req=0xc52f60
2009/12/15 17:09:15| clientProcessBody: start fd=35 body_size=0 in.offset=0 
cb=0x4562cd req=0xc52f60
2009/12/15 17:09:15| clientReadRequest: FD 35: reading request...
2009/12/15 17:09:15| clientReadRequest: FD 35: read 311 bytes
2009/12/15 17:09:15| clientProcessBody: start fd=35 body_size=0 in.offset=311 
cb=0x4562cd req=0xc52f60
2009/12/15 17:09:15| assertion failed: client_side.c:4471: 
"conn->body.size_left > 0"

am I on the right track here?

--
Mark Nottingham   m...@yahoo-inc.com

Hudson build is back to normal: 3.HEAD-amd64-CentOS-5.3 #280

[noc]

See