---------- Forwarded message ----------
Date: Tue, 17 May 2005 13:56:23 +1200
From: "Beamish-White, James, VF-NZ" <[EMAIL PROTECTED]>
To: Henrik Nordstrom <[EMAIL PROTECTED]>
Subject: RE: Modifying User-Agent
Attached is the patch (generated from diff -Naur).
The main changes are made in HttpHeaderTools.c, but the error I am
getting is:
helperHandleRead: unexpected read from header_rewriter #1, 161 bytes
...which is debug in the helper.c file.
Thanks!
James
-----Original Message-----
From: Henrik Nordstrom [mailto:[EMAIL PROTECTED]
Sent: Tuesday, 17 May 2005 12:58 p.m.
To: Beamish-White, James, VF-NZ
Cc: squid-dev@squid-cache.org
Subject: Re: Modifying User-Agent
On Tue, 17 May 2005, Beamish-White, James, VF-NZ wrote:
I have got some patch code that another developer did 4 years ago,
which I'm trying to get to work against the current stable release of
squid.
Ok.
What we need to do is use squid to pass the User-Agent header to a
helper (e.g. perl script) to rewrite as necessary. This is in
particular to strip a -EMULATOR off phone names (e.g. User-Agent:
SHARP-G3-EMULATOR) for a WAP/3G service. I have it mostly working, but
there's an error when it does work, and if I also include the
redirect_program directive in the squid.conf (a requirement), it hangs
without response.
If you could post the patch it would be easier to help you out.
From your problem description it sounds like the patch hooks into the
request flow in a sub-optimal manner..
If not done already I would probably add a lookup cache as well to avoid
repeated helper lookups for the same user-agent string.
Regards
Henrik
-----------------------------------------------------------------------------------------------
Have you seen our website?.... http://www.vodafone.co.nz
Manage Your Account, check your Vodafone Mail and send web2TXT online:
http://www.vodafone.co.nz/myvodafone
CAUTION: This correspondence is confidential and intended for the named
recipient(s) only.
If you are not the named recipient and receive this correspondence in error,
you must not copy,
distribute or take any action in reliance on it and you should delete it from
your system and
notify the sender immediately. Thank you.
Unless otherwise stated, any views or opinions expressed are solely those of
the author and do
not represent those of Vodafone New Zealand Limited.
Vodafone New Zealand Limited
20 Viaduct Harbour Avenue, Private Bag 92161, Auckland 1030
Telephone + 64 9 355 2000
Facsimile + 64 9 355 2001
diff --exclude=.svn -Naur squid-2.5.STABLE9/src/cache_cf.c
squid-2.5.james01/src/cache_cf.c
--- squid-2.5.STABLE9/src/cache_cf.c 2005-05-17 13:43:53.803459848 +1200
+++ squid-2.5.james01/src/cache_cf.c 2005-05-16 13:44:21.000000000 +1200
@@ -312,6 +312,13 @@
wordlistDestroy(&Config.Program.redirect);
}
}
+ /* JBW: Add header_rewrite.command entry */
+ if (Config.Program.header_rewrite.command) {
+ if (Config.Program.header_rewrite.children < 1) {
+ Config.Program.header_rewrite.children = 0;
+ wordlistDestroy(&Config.Program.header_rewrite.command);
+ }
+ }
if (Config.Accel.host) {
snprintf(buf, BUFSIZ, "http://%s:%d";, Config.Accel.host,
Config.Accel.port);
Config2.Accel.prefix = xstrdup(buf);
@@ -365,6 +372,9 @@
#endif
if (Config.Program.redirect)
requirePathnameExists("redirect_program", Config.Program.redirect->key);
+ /* JBW: Add header_rewrite section */
+ if (Config.Program.header_rewrite.command)
+ requirePathnameExists("header_rewrite_program",
Config.Program.header_rewrite.command->key);
requirePathnameExists("Icon Directory", Config.icons.directory);
requirePathnameExists("Error Directory", Config.errorDirectory);
#if HTTP_VIOLATIONS
diff --exclude=.svn -Naur squid-2.5.STABLE9/src/cf.data
squid-2.5.james01/src/cf.data
--- squid-2.5.STABLE9/src/cf.data 2005-05-17 13:43:53.819457416 +1200
+++ squid-2.5.james01/src/cf.data 2005-05-16 13:44:21.000000000 +1200
@@ -1264,6 +1264,55 @@
are sent.
DOC_END
+# JBW: Added
header_rewrite_program,header_rewrite_children,header_rewrite_access
+NAME: header_rewrite_program
+TYPE: wordlist
+LOC: Config.Program.header_rewrite.command
+DEFAULT: none
+DOC_START
+ Specify the location of the executable for the header rewriter.
+ Since they can perform almost any function there isn't one included.
+ For each requested URL which matches the header_rewriter
+ ACL, Squid will send a block of header lines to the rewriter program
terminated
+ by an empty line :
+
+
+ headerl: headerl_contents
+ header2: header2_contents
+ header3: header3_contents
+ header4: header4 contents
+ \n
+
+ And the rewriter should return modified or additional headers. Note
+ that all of the original request headers will be removed and replaced
+ by whatever the header_rewriter generates.
+
+ By default, a header rewriter is not used.
+DOC_END
+
+NAME: header_rewrite_children
+TYPE: int
+DEFAULT: 5
+LOC: Config.Program.header_rewrite.children
+DOC_START
+ The number of header rewriting processes to spawn. If you start
+ too few Squid will have to wait for them to process a backlog of
+ headers, slowing it down. If you start too many they will use RAM
+ and other system resources.
+DOC_END
+
+NAME: header_rewrite_access
+TYPE: acl_access
+DEFAULT: none
+LOC: Config.accessList.header_rewrite
+DOC_START
+ If defined, this access list specifies which requests are
+ sent to the header rewriting processes. By default all requests
+ are sent.
+DOC_END
+
+
+
NAME: auth_param
TYPE: authparam
LOC: Config.authConfig
diff --exclude=.svn -Naur squid-2.5.STABLE9/src/cf_parser.h
squid-2.5.james01/src/cf_parser.h
--- squid-2.5.STABLE9/src/cf_parser.h 2005-05-17 13:43:53.771464712 +1200
+++ squid-2.5.james01/src/cf_parser.h 2005-05-16 16:52:29.000000000 +1200
@@ -106,6 +106,9 @@
default_line("redirect_children 5");
default_line("redirect_rewrites_host_header on");
/* No default for redirector_access */
+ /* No default for header_rewrite_program */
+ default_line("header_rewrite_children 5");
+ /* No default for header_rewrite_access */
/* No default for auth_param */
default_line("authenticate_cache_garbage_interval 1 hour");
default_line("authenticate_ttl 1 hour");
@@ -520,6 +523,12 @@
parse_onoff(&Config.onoff.redir_rewrites_host);
else if (!strcmp(token, "redirector_access"))
parse_acl_access(&Config.accessList.redirector);
+ else if (!strcmp(token, "header_rewrite_program"))
+ parse_wordlist(&Config.Program.header_rewrite.command);
+ else if (!strcmp(token, "header_rewrite_children"))
+ parse_int(&Config.Program.header_rewrite.children);
+ else if (!strcmp(token, "header_rewrite_access"))
+ parse_acl_access(&Config.accessList.header_rewrite);
else if (!strcmp(token, "auth_param"))
parse_authparam(&Config.authConfig);
else if (!strcmp(token, "authenticate_cache_garbage_interval"))
@@ -965,6 +974,9 @@
dump_int(entry, "redirect_children", Config.redirectChildren);
dump_onoff(entry, "redirect_rewrites_host_header",
Config.onoff.redir_rewrites_host);
dump_acl_access(entry, "redirector_access",
Config.accessList.redirector);
+ dump_wordlist(entry, "header_rewrite_program",
Config.Program.header_rewrite.command);
+ dump_int(entry, "header_rewrite_children",
Config.Program.header_rewrite.children);
+ dump_acl_access(entry, "header_rewrite_access",
Config.accessList.header_rewrite);
dump_authparam(entry, "auth_param", Config.authConfig);
dump_time_t(entry, "authenticate_cache_garbage_interval",
Config.authenticateGCInterval);
dump_time_t(entry, "authenticate_ttl", Config.authenticateTTL);
@@ -1257,6 +1269,9 @@
free_int(&Config.redirectChildren);
free_onoff(&Config.onoff.redir_rewrites_host);
free_acl_access(&Config.accessList.redirector);
+ free_wordlist(&Config.Program.header_rewrite.command);
+ free_int(&Config.Program.header_rewrite.children);
+ free_acl_access(&Config.accessList.header_rewrite);
free_authparam(&Config.authConfig);
free_time_t(&Config.authenticateGCInterval);
free_time_t(&Config.authenticateTTL);
diff --exclude=.svn -Naur squid-2.5.STABLE9/src/client_side.c
squid-2.5.james01/src/client_side.c
--- squid-2.5.STABLE9/src/client_side.c 2005-05-17 13:43:53.794461216 +1200
+++ squid-2.5.james01/src/client_side.c 2005-05-11 10:42:02.000000000 +1200
@@ -398,7 +398,9 @@
clientHttpRequest *http = data;
http->request->flags.cachable = answer;
http->acl_checklist = NULL;
- clientProcessRequest(http);
+ /* JBW: Commented : "clientProcessRequest(http);" */
+ /* KBT+JBW: call header rewriter */
+ hdrRewriteStart(http, clientProcessRequest, http );
}
static void
diff --exclude=.svn -Naur squid-2.5.STABLE9/src/helper.c
squid-2.5.james01/src/helper.c
--- squid-2.5.STABLE9/src/helper.c 2005-05-17 13:43:53.000000000 +1200
+++ squid-2.5.james01/src/helper.c 2005-05-17 13:36:11.000000000 +1200
@@ -669,10 +669,21 @@
debug(84, 1) ("helperHandleRead: unexpected read from %s #%d, %d
bytes\n",
hlp->id_name, srv->index + 1, len);
srv->offset = 0;
- } else if ((t = strchr(srv->buf, '\n'))) {
- /* end of reply found */
- debug(84, 3) ("helperHandleRead: end of reply found\n");
- *t = '\0';
+
+ /* KBT+JBW : below changed to look for \n\n if double_cr_end is set */
+ } else if ((hlp->double_cr_end && (t = strchr(srv->buf, '\n'))) ||
+ (hlp->double_cr_end && (t = strstr(srv->buf, "\r\n\r\n")))
+ ) { /* end of reply found */
+ debug(84, 3) ("helperHandleRead: end of reply found\n");
+ if( strstr(srv->buf, "\r\n\r\n") )
+ t += 2; *t = '\0';
+
+ /* JBW: Comments for cutting out diff
+ // if (cbdataValid(r->data))
+ // r->callback(r->data, srv->buf);
+ // end of reply found
+ // debug(84, 3) ("helperHandleRead: end of reply found\n");
+ // *t = '\0'; */
srv->flags.busy = 0;
srv->offset = 0;
srv->request = NULL;
diff --exclude=.svn -Naur squid-2.5.STABLE9/src/HttpHeaderTools.c
squid-2.5.james01/src/HttpHeaderTools.c
--- squid-2.5.STABLE9/src/HttpHeaderTools.c 2005-05-17 13:43:53.000000000
+1200
+++ squid-2.5.james01/src/HttpHeaderTools.c 2005-05-16 13:44:21.000000000
+1200
@@ -469,3 +469,220 @@
if (0 == httpHdrMangle(e, request))
httpHeaderDelAt(l, p);
}
+
+/* *** KBT+JBW : Start of header rewriter helper code *** */
+
+
+#define HDR_BUFSIZE 8192
+
+typedef struct {
+ void *data;
+ char *orig_url; /* Copy of original URL */
+ struct in_addr client_addr;
+ const char *client_ident;
+ RH *handler;
+ HttpHeader *req_header; /* Pointer to request headers */
+} hdrHelper_StateData;
+
+static HLPCB hdrHandleReply;
+static helper *hdrHelpers = NULL;
+static OBJH hdrHelperStats;
+static int hdrHelper_n_bypassed = 0;
+CBDATA_TYPE(hdrHelper_StateData);
+
+static void
+hdrStateFree(hdrHelper_StateData *r)
+ {
+ safe_free(r->orig_url);
+ cbdataFree(r);
+ }
+/*
+ hdrHandleReply
+ function which handles the replies from the header rewriter helpers
+
+ read the reply from the handler, and add it to the rewritten_headers
+ in new headers array
+ if the reply is END, then :
+ - remove the headers from the request
+ - copy the rewritten headers into the request
+ - call the handler ( ie, return )
+*/
+static void
+hdrHandleReply(void *data, char *reply)
+{
+ hdrHelper_StateData *r = data;
+ int valid;
+
+ debug(66, 5) ("hdrHandleReply: {%s}\n", reply ? reply : "<NULL>");
+ if (reply) {
+ debug(66, 5) ("hdrHandleReply: doing header parse\n");
+ /* parse the reply in separate lines and insert each line into header
entries */
+ httpHeaderReset( r->req_header );
+ httpHeaderParse( r->req_header, reply, reply + strlen(reply) );
+ debug(66, 5) ("hdrHandleReply: done header parse\n");
+ }
+
+ /*
+ End of helper request, call the callback function
+ */
+ valid = cbdataValid(r->data);
+ cbdataUnlock(r->data);
+ debug(66, 3) ("hdrHandleReply: rewrite complete, calling handler\n");
+ if (valid)
+ r->handler(r->data, reply);
+ hdrStateFree(r);
+}
+
+static void
+hdrHelperStats(StoreEntry * sentry)
+{
+ storeAppendPrintf(sentry, "Header Rewriter Statistics:\n");
+ helperStats(sentry, hdrHelpers);
+ storeAppendPrintf(sentry, "\nNumber of requests bypassed "
+ "because all rewriters were busy: %d\n", hdrHelper_n_bypassed);
+}
+
+/*
+ public function hdrRewritelnit()
+ Startup the header rewriter helpers ( if configured )
+ */
+void
+hdrRewritelnit(void) {
+ static int init = 0;
+ debug(66,2) ("hdrRewritelnit:\n");
+ if (!Config.Program.header_rewrite.command){
+ debug(66,2) ("hdrRewritelnit: quitting - no header_rewrite command\n");
+ return;
+ }
+ if (hdrHelpers == NULL){
+ debug(66,2) ("hdrRewritelnit: creating helpers\n");
+ hdrHelpers = helperCreate("header_rewriter");
+ }
+ hdrHelpers->cmdline = Config. Program.header_rewrite.command;
+ hdrHelpers->n_to_start = Config.Program.header_rewrite.children;
+ hdrHelpers->ipc_type = IPC_TCP_SOCKET;
+ hdrHelpers->double_cr_end = 1;
+ helperOpenServers(hdrHelpers);
+ if (!init) {
+ cachemgrRegister("header_rewriter", "Header Rewriter Stats",
hdrHelperStats, 0, 1);
+ init = 1;
+ CBDATA_INIT_TYPE(hdrHelper_StateData);
+ }
+ debug(66,2) ("hdrRewritelnit: done\n");
+}
+
+
+/*
+ public function hdrRewriteShutdown()
+
+ Shutdown the header rewriter helpers ( if configured )
+*/
+void
+hdrRewriteShutdown() {
+ debug(66,2) ("hdrRewriteShutdown: starting-VI");
+ if (!hdrHelpers)
+ return;
+ helperShutdown(hdrHelpers);
+ if (!shutting_down)
+ return;
+ helperFree(hdrHelpers);
+ hdrHelpers = NULL;
+ debug(66,2) ("hdrRewriteShutdown: done\n");
+}
+
+/*
+ public function hdrRewriteStart()
+ Call the header rewriter helpers
+
+ *** This needs serious fixing ***
+
+*/
+void
+hdrRewriteStart(clientHttpRequest * http, RH * handler, void *data)
+{
+ ConnStateData *conn = http->conn;
+ hdrHelper_StateData *helper = NULL;
+ char buf[HDR_BUFSIZE], *cp;
+ HttpHeader *reqheaders = &http->request->header;
+ HttpHeaderEntry *e;
+ HttpHeaderPos p = HttpHeaderInitPos;
+ int nl, vl;
+
+ assert(http);
+ assert(handler);
+ debug(66, 3) ("hdrRewriteStart: '%s'\n", http->uri);
+ if (Config.Program.header_rewrite.command == NULL) {
+ /* No header rewriting command has been defined, so just return by
calling the handler
+ */
+ debug(66, 3) ("hdrRewriteStart: finishing because header_rewrite command
not defined\n");
+ handler(data, NULL);
+ return;
+ }
+
+ if (Config.accessList.header_rewrite) {
+ aclCheck_t ch;
+ memset(&ch, '\0', sizeof(ch));
+ ch.src_addr = http->conn->peer.sin_addr;
+ ch.my_addr = http->conn->me.sin_addr;
+ ch.my_port = ntohs(http->conn->me.sin_port);
+ ch.request = http->request;
+ if (!aclCheckFast(Config.accessList.header_rewrite, &ch)) {
+ /* denied -- bypass redirector */
+
+ debug(66, 3) ("hdrRewriteStart: finishing because request not in
ACL\n");
+ handler(data, NULL);
+ return;
+ }
+ }
+
+ if (hdrHelpers->stats.queue_size) {
+ /* Skip helper if there is one request queued */
+ hdrHelper_n_bypassed++;
+ debug(66, 3) ("hdrRewriteStart: finishing because too many requests
queued\n");
+ handler(data, NULL);
+ return;
+ }
+
+ /*
+ put together the callback struct
+ */
+ helper = cbdataAlloc(hdrHelper_StateData);
+ helper->orig_url = xstrdup(http->uri);
+ helper->client_addr = conn->log_addr;
+ if (http->request->auth_user_request)
+ helper->client_ident =
authenticateUserRequestUsername(http->request->auth_user_request);
+ else if (conn->rfc931[0]) {
+ helper->client_ident = conn->rfc931;
+ } else {
+ helper->client_ident = dash_str;
+ }
+ helper->handler = handler;
+ helper->data = data;
+ cbdataLock(helper->data);
+ helper->req_header = reqheaders;
+
+ /*
+ Format the buffer in preparation for sending to helper
+ */
+ cp = buf;
+ while ((e = httpHeaderGetEntry(reqheaders, &p))){
+ debug(66, 3) ("hdrRewriteStart: sending header
%s:%s\n",strBuf(e->name),strBuf(e->value));
+ nl = strlen(strBuf(e->name));
+ vl = strlen(strBuf(e->value));
+ assert( (cp + nl + vl + 4) < (buf + HDR_BUFSIZE) );
+ strcpy(cp,strBuf(e->name)); cp += nl;
+ strcpy(cp,": "); cp += 2;
+ strcpy(cp,strBuf(e->value)); cp += vl;
+ strcpy(cp,"\r\n"); cp += 2;
+ }
+ /* Extra \n to terminate */
+ strcpy(cp,"\r\n");
+
+ helperSubmit(hdrHelpers, buf, hdrHandleReply, helper);
+
+
+}
+
+/* *** End of KBT+JBW header rewriter helper code *** */
+
+
diff --exclude=.svn -Naur squid-2.5.STABLE9/src/main.c
squid-2.5.james01/src/main.c
--- squid-2.5.STABLE9/src/main.c 2005-05-17 13:43:53.000000000 +1200
+++ squid-2.5.james01/src/main.c 2005-05-11 10:42:02.000000000 +1200
@@ -349,6 +349,8 @@
idnsShutdown();
#endif
redirectShutdown();
+ /* JBW: add hdrRewriteShutdown(); */
+ hdrRewriteShutdown();
authenticateShutdown();
externalAclShutdown();
storeDirCloseSwapLogs();
@@ -376,6 +378,8 @@
idnsInit();
#endif
redirectInit();
+ /* JBW: add hdrRewritelnit(); */
+ hdrRewritelnit();
authenticateInit(&Config.authConfig);
externalAclInit();
#if USE_WCCP
@@ -404,6 +408,8 @@
dnsShutdown();
#endif
redirectShutdown();
+ /* JBW: Add hdrRewriteShutdown(); */
+ hdrRewriteShutdown();
authenticateShutdown();
externalAclShutdown();
_db_rotate_log(); /* cache.log */
@@ -420,6 +426,8 @@
dnsInit();
#endif
redirectInit();
+ /* JBW: Add hdrRewritelnit(); */
+ hdrRewritelnit();
authenticateInit(&Config.authConfig);
externalAclInit();
}
@@ -505,6 +513,8 @@
idnsInit();
#endif
redirectInit();
+ /* JBW: Add hdrRewritelnit(); */
+ hdrRewritelnit();
authenticateInit(&Config.authConfig);
externalAclInit();
useragentOpenLog();
@@ -730,6 +740,8 @@
do_shutdown = 0;
shutting_down = 1;
serverConnectionsClose();
+ /* JBW: Add hdrRewriteShutdown(); */
+ hdrRewriteShutdown();
eventAdd("SquidShutdown", SquidShutdown, NULL, (double) (wait + 1),
1);
}
eventRun();
diff --exclude=.svn -Naur squid-2.5.STABLE9/src/protos.h
squid-2.5.james01/src/protos.h
--- squid-2.5.STABLE9/src/protos.h 2005-05-17 13:43:53.000000000 +1200
+++ squid-2.5.james01/src/protos.h 2005-05-11 10:42:02.000000000 +1200
@@ -416,6 +416,10 @@
#else
extern void httpHeaderPutStrf();
#endif
+/* JBW: Add hdrRewritelnit, hdrRewriteShutdown, hdrRewriteStart */
+extern void hdrRewritelnit();
+extern void hdrRewriteShutdown();
+extern void hdrRewriteStart(clientHttpRequest*, RH *, void *);
/* Http Header */
diff --exclude=.svn -Naur squid-2.5.STABLE9/src/squid.conf.default
squid-2.5.james01/src/squid.conf.default
--- squid-2.5.STABLE9/src/squid.conf.default 2005-05-17 13:43:53.000000000
+1200
+++ squid-2.5.james01/src/squid.conf.default 2005-05-16 16:52:29.000000000
+1200
@@ -1052,6 +1052,46 @@
#Default:
# none
+# TAG: header_rewrite_program
+# Specify the location of the executable for the header rewriter.
+# Since they can perform almost any function there isn't one included.
+# For each requested URL which matches the header_rewriter
+# ACL, Squid will send a block of header lines to the rewriter program
terminated
+# by an empty line :
+#
+#
+# headerl: headerl_contents
+# header2: header2_contents
+# header3: header3_contents
+# header4: header4 contents
+# \n
+#
+# And the rewriter should return modified or additional headers. Note
+# that all of the original request headers will be removed and replaced
+# by whatever the header_rewriter generates.
+#
+# By default, a header rewriter is not used.
+#
+#Default:
+# none
+
+# TAG: header_rewrite_children
+# The number of header rewriting processes to spawn. If you start
+# too few Squid will have to wait for them to process a backlog of
+# headers, slowing it down. If you start too many they will use RAM
+# and other system resources.
+#
+#Default:
+# header_rewrite_children 5
+
+# TAG: header_rewrite_access
+# If defined, this access list specifies which requests are
+# sent to the header rewriting processes. By default all requests
+# are sent.
+#
+#Default:
+# none
+
# TAG: auth_param
# This is used to define parameters for the various authentication
# schemes supported by Squid.
diff --exclude=.svn -Naur squid-2.5.STABLE9/src/structs.h
squid-2.5.james01/src/structs.h
--- squid-2.5.STABLE9/src/structs.h 2005-05-17 13:43:53.000000000 +1200
+++ squid-2.5.james01/src/structs.h 2005-05-11 10:42:02.000000000 +1200
@@ -120,6 +120,12 @@
auth_user_hash_pointer *usernamehash;
/* cache of acl lookups on this username */
dlink_list proxy_match_cache;
+
+ /*0=unchecked,l=ok,2=failed */
+ struct {
+ unsigned int credentials:1;
+ } flags;
+
/* what ip addresses has this user been seen at?, plus a list length cache
*/
dlink_list ip_list;
size_t ipcount;
@@ -486,6 +492,13 @@
char *dnsserver;
#endif
wordlist *redirect;
+ /* JBW: Add header_rewrite and url_rewrite */
+ struct {
+ wordlist *command;
+ int children;
+ } header_rewrite;
+ acl_access *url_rewrite;
+
#if USE_ICMP
char *pinger;
#endif
@@ -626,6 +639,8 @@
acl_access *identLookup;
#endif
acl_access *redirector;
+ /* JBW: Add header rewrite */
+ acl_access *header_rewrite;
acl_access *reply;
acl_address *outgoing_address;
acl_tos *outgoing_tos;
@@ -2033,6 +2048,8 @@
int n_running;
int n_active;
int ipc_type;
+ /* JBW: add double_cr_end */
+ int double_cr_end;
time_t last_queue_warn;
struct {
int requests;
