On 9/5/2012 8:00 AM, Amos Jeffries wrote:
On 5/09/2012 4:10 p.m., Eliezer Croitoru wrote:
I'm reading some code (will take a while) to maybe get a functional
store_url_rewrite for squid3+.
Actually i was thinking about it a lot and the process should be very
simple:
use some stdin+stdout like for url_rewrite interface for starter.
Yes. The redirect.cc interface can be used. The caller code runs
*_access checks and provides a callback function which uses the
resulting URL to do things to store keys.
i think this is can be done pretty fast if someone knows the current
code.
it's kind of replicate the url_rewrite and change all the directives
names from url_rewrite to store_url_rewrite and let someone write the
functions.
how is this as a starter to maybe make it work?
Exactly what I was planning. Just a lack of store code knowledge getting
in the way.
It should be done before the request is being done and i think that
before url_rewrite helper.
At the point where cache key is being decided I think. Which is just
before HIT lookup. After adaptations like url_rewrite.
the next step after that is to take the "to be cached" entry and
change the the key of the stored object. as it will be stored at in
the cache and not like the old store_url_rewrite that was saved in a
formart of old+new url.
how about just start with the basics of putting the whole redirector
as in with store_url_rewrite?
how hard can it be?
Want to give it a try?
Well I would like to but it seems i'm lacking of knowledge in c\c++ and
squid structure.
I do recognize that there is a language there with some structure since
I did wrote using java and ruby.
(just wrote a small proxy for specific protocol that peers two tcp
incoming connections after verification of identity and it seems to rock)
I was thinking to start with creating a store_url_rewrite (fake) that
will debug and all but will not do any changes except logging.
If I\we can make this stage it will benefit anyone who will want to
integrate some new interface for squid.
So I started playing with the sources but since squid is a big puzzle
for me I dont know where and what I should put or look for pieces of
code related to the redirect.cc .
what I did as a starter is to take redirect.cc (3.2.1)and change all the
methods names I could find\think off to a "srewriter" syntax such as:
redirectRegisterWithCacheManager();
to
srewriteRegisterWithCacheManager();
file and diff attached.
after that I tried to find anything related to redirect.cc in the
makefiles and have seen:
at:
./src/Makefile.am
./src/Makefile.in
so i suppose i should add to every place of redirect.cc also srewrite.cc
(but i'm dont know...)
I didnt made any changed to the srewrite.cc that will make it "harmless"
and "fake" yet.
now the hard part for me is: how would I find any redirect.cc in the
rest of squid and make sure to add the needed code for srewrite.cc?
I can parse the srewrite.cc but since I dont know any of the code
structure I wouldnt know what to extract and look for.
I noticed kinkie did pretty things while parsing the code.
I took a peer at the old store_url_rewrite and it seems to be divided
into "client_side_storeurl_rewrite.c", "store_rewrite.c"
and has some struct in structs.h for children concurrency and command.
On 3.2.1 it's seems to be organized better and only in couple files such
as the redirect.cc and others.
any leads,?
Eliezer
--
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer <at> ngtech.co.il
#diff redirect.cc srewriter.cc
5c5
< * DEBUG: section 61 Redirector
---
> * DEBUG: section 61 Srewriter
55c55
< /// url maximum lengh + extra informations passed to redirector
---
> /// url maximum lengh + extra informations passed to srewriter
66c66
< } redirectStateData;
---
> } srewriterStateData;
68,71c68,71
< static HLPCB redirectHandleReply;
< static void redirectStateFree(redirectStateData * r);
< static helper *redirectors = NULL;
< static OBJH redirectStats;
---
> static HLPCB srewriteHandleReply;
> static void srewriterStateFree(srewriterStateData * r);
> static helper *srewriters = NULL;
> static OBJH srewriterStats;
73c73
< CBDATA_TYPE(redirectStateData);
---
> CBDATA_TYPE(srewriterStateData);
76c76
< redirectHandleReply(void *data, char *reply)
---
> srewriteHandleReply(void *data, char *reply)
78c78
< redirectStateData *r = static_cast<redirectStateData *>(data);
---
> srewriterStateData *r = static_cast<srewriterStateData *>(data);
81c81
< debugs(61, 5, "redirectHandleRead: {" << (reply && *reply != '\0' ? reply
: "<NULL>") << "}");
---
> debugs(61, 5, "srewriteHandleRead: {" << (reply && *reply != '\0' ? reply
> : "<NULL>") << "}");
94c94
< redirectStateFree(r);
---
> srewriterStateFree(r);
98c98
< redirectStateFree(redirectStateData * r)
---
> srewriterStateFree(srewriterStateData * r)
105c105
< redirectStats(StoreEntry * sentry)
---
> srewriterStats(StoreEntry * sentry)
107,108c107,108
< if (redirectors == NULL) {
< storeAppendPrintf(sentry, "No redirectors defined\n");
---
> if (srewriters == NULL) {
> storeAppendPrintf(sentry, "No srewriters defined\n");
112c112
< helperStats(sentry, redirectors, "Redirector Statistics");
---
> helperStats(sentry, srewriters, "Srewriter Statistics");
114c114
< if (Config.onoff.redirector_bypass)
---
> if (Config.onoff.srewriter_bypass)
116c116
< "because all redirectors were busy: %d\n",
n_bypassed);
---
> "because all srewriters were busy: %d\n",
> n_bypassed);
122c122
< redirectStart(ClientHttpRequest * http, RH * handler, void *data)
---
> srewriterStart(ClientHttpRequest * http, RH * handler, void *data)
125c125
< redirectStateData *r = NULL;
---
> srewriterStateData *r = NULL;
134c134
< debugs(61, 5, "redirectStart: '" << http->uri << "'");
---
> debugs(61, 5, "srewriterStart: '" << http->uri << "'");
136,137c136,137
< if (Config.onoff.redirector_bypass && redirectors->stats.queue_size) {
< /* Skip redirector if there is one request queued */
---
> if (Config.onoff.srewriter_bypass && srewriters->stats.queue_size) {
> /* Skip srewriter if there is one request queued */
143c143
< r = cbdataAlloc(redirectStateData);
---
> r = cbdataAlloc(srewriterStateData);
200c200
< debugs(61, DBG_CRITICAL, "ERROR: Gateway Failure. Can not build
request to be passed to redirector. Request ABORTED.");
---
> debugs(61, DBG_CRITICAL, "ERROR: Gateway Failure. Can not build
> request to be passed to srewriter. Request ABORTED.");
203c203
< debugs(61, DBG_CRITICAL, "ERROR: Gateway Failure. Request passed
to redirector exceeds MAX_REDIRECTOR_REQUEST_STRLEN (" <<
MAX_REDIRECTOR_REQUEST_STRLEN << "). Request ABORTED.");
---
> debugs(61, DBG_CRITICAL, "ERROR: Gateway Failure. Request passed
> to srewriter exceeds MAX_REDIRECTOR_REQUEST_STRLEN (" <<
> MAX_REDIRECTOR_REQUEST_STRLEN << "). Request ABORTED.");
230c230
< helperSubmit(redirectors, buf, redirectHandleReply, r);
---
> helperSubmit(srewriters, buf, srewriteHandleReply, r);
234c234
< redirectRegisterWithCacheManager(void)
---
> srewriteRegisterWithCacheManager(void)
236c236
< Mgr::RegisterAction("redirector", "URL Redirector Stats", redirectStats,
0, 1);
---
> Mgr::RegisterAction("srewriter", "URL Srewriter Stats", srewriterStats,
> 0, 1);
240c240
< redirectInit(void)
---
> srewriteInit(void)
244c244
< redirectRegisterWithCacheManager();
---
> srewriteRegisterWithCacheManager();
246c246
< if (!Config.Program.redirect)
---
> if (!Config.Program.srewrite)
249,250c249,250
< if (redirectors == NULL)
< redirectors = new helper("redirector");
---
> if (srewriters == NULL)
> srewriters = new helper("srewriter");
252c252
< redirectors->cmdline = Config.Program.redirect;
---
> srewriters->cmdline = Config.Program.srewrite;
254c254
< redirectors->childs.updateLimits(Config.redirectChildren);
---
> srewriters->childs.updateLimits(Config.srewriteChildren);
256c256
< redirectors->ipc_type = IPC_STREAM;
---
> srewriters->ipc_type = IPC_STREAM;
258c258
< helperOpenServers(redirectors);
---
> helperOpenServers(srewriters);
262c262
< CBDATA_INIT_TYPE(redirectStateData);
---
> CBDATA_INIT_TYPE(srewriterStateData);
267c267
< redirectShutdown(void)
---
> srewriterShutdown(void)
269c269
< if (!redirectors)
---
> if (!srewriters)
272c272
< helperShutdown(redirectors);
---
> helperShutdown(srewriters);
277,278c277,278
< delete redirectors;
< redirectors = NULL;
---
> delete srewriters;
> srewriters = NULL;/*
* $Id$
*
* DEBUG: section 61 Srewriter
* AUTHOR: Duane Wessels
*
* SQUID Web Proxy Cache http://www.squid-cache.org/
* ----------------------------------------------------------
*
* Squid is the result of efforts by numerous individuals from
* the Internet community; see the CONTRIBUTORS file for full
* details. Many organizations have provided support for Squid's
* development; see the SPONSORS file for full details. Squid is
* Copyrighted (C) 2001 by the Regents of the University of
* California; see the COPYRIGHT file for full details. Squid
* incorporates software developed and/or copyrighted by other
* sources; see the CREDITS file for full details.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111, USA.
*
*/
#include "squid-old.h"
#if USE_AUTH
#include "auth/UserRequest.h"
#endif
#include "comm/Connection.h"
#include "mgr/Registration.h"
#include "Store.h"
#include "fde.h"
#include "client_side_request.h"
#include "acl/Checklist.h"
#include "HttpRequest.h"
#include "client_side.h"
#include "client_side_reply.h"
#include "helper.h"
#include "rfc1738.h"
#if USE_SSL
#include "ssl/support.h"
#endif
/// url maximum lengh + extra informations passed to srewriter
#define MAX_REDIRECTOR_REQUEST_STRLEN (MAX_URL + 1024)
typedef struct {
void *data;
char *orig_url;
Ip::Address client_addr;
const char *client_ident;
const char *method_s;
RH *handler;
} srewriterStateData;
static HLPCB srewriteHandleReply;
static void srewriterStateFree(srewriterStateData * r);
static helper *srewriters = NULL;
static OBJH srewriterStats;
static int n_bypassed = 0;
CBDATA_TYPE(srewriterStateData);
static void
srewriteHandleReply(void *data, char *reply)
{
srewriterStateData *r = static_cast<srewriterStateData *>(data);
char *t;
void *cbdata;
debugs(61, 5, "srewriteHandleRead: {" << (reply && *reply != '\0' ? reply :
"<NULL>") << "}");
if (reply) {
if ((t = strchr(reply, ' ')))
*t = '\0';
if (*reply == '\0')
reply = NULL;
}
if (cbdataReferenceValidDone(r->data, &cbdata))
r->handler(cbdata, reply);
srewriterStateFree(r);
}
static void
srewriterStateFree(srewriterStateData * r)
{
safe_free(r->orig_url);
cbdataFree(r);
}
static void
srewriterStats(StoreEntry * sentry)
{
if (srewriters == NULL) {
storeAppendPrintf(sentry, "No srewriters defined\n");
return;
}
helperStats(sentry, srewriters, "Srewriter Statistics");
if (Config.onoff.srewriter_bypass)
storeAppendPrintf(sentry, "\nNumber of requests bypassed "
"because all srewriters were busy: %d\n", n_bypassed);
}
/**** PUBLIC FUNCTIONS ****/
void
srewriterStart(ClientHttpRequest * http, RH * handler, void *data)
{
ConnStateData * conn = http->getConn();
srewriterStateData *r = NULL;
const char *fqdn;
char buf[MAX_REDIRECTOR_REQUEST_STRLEN];
int sz;
http_status status;
char claddr[MAX_IPSTRLEN];
char myaddr[MAX_IPSTRLEN];
assert(http);
assert(handler);
debugs(61, 5, "srewriterStart: '" << http->uri << "'");
if (Config.onoff.srewriter_bypass && srewriters->stats.queue_size) {
/* Skip srewriter if there is one request queued */
++n_bypassed;
handler(data, NULL);
return;
}
r = cbdataAlloc(srewriterStateData);
r->orig_url = xstrdup(http->uri);
if (conn != NULL)
r->client_addr = conn->log_addr;
else
r->client_addr.SetNoAddr();
r->client_ident = NULL;
#if USE_AUTH
if (http->request->auth_user_request != NULL) {
r->client_ident = http->request->auth_user_request->username();
debugs(61, 5, HERE << "auth-user=" <<
(r->client_ident?r->client_ident:"NULL"));
}
#endif
// HttpRequest initializes with null_string. So we must check both
defined() and size()
if (!r->client_ident && http->request->extacl_user.defined() &&
http->request->extacl_user.size()) {
r->client_ident = http->request->extacl_user.termedBuf();
debugs(61, 5, HERE << "acl-user=" <<
(r->client_ident?r->client_ident:"NULL"));
}
if (!r->client_ident && conn != NULL && conn->clientConnection != NULL &&
conn->clientConnection->rfc931[0]) {
r->client_ident = conn->clientConnection->rfc931;
debugs(61, 5, HERE << "ident-user=" <<
(r->client_ident?r->client_ident:"NULL"));
}
#if USE_SSL
if (!r->client_ident && conn != NULL &&
Comm::IsConnOpen(conn->clientConnection)) {
r->client_ident =
sslGetUserEmail(fd_table[conn->clientConnection->fd].ssl);
debugs(61, 5, HERE << "ssl-user=" <<
(r->client_ident?r->client_ident:"NULL"));
}
#endif
if (!r->client_ident)
r->client_ident = dash_str;
r->method_s = RequestMethodStr(http->request->method);
r->handler = handler;
r->data = cbdataReference(data);
if ((fqdn = fqdncache_gethostbyaddr(r->client_addr, 0)) == NULL)
fqdn = dash_str;
sz = snprintf(buf, MAX_REDIRECTOR_REQUEST_STRLEN, "%s %s/%s %s %s myip=%s
myport=%d\n",
r->orig_url,
r->client_addr.NtoA(claddr,MAX_IPSTRLEN),
fqdn,
r->client_ident[0] ? rfc1738_escape(r->client_ident) :
dash_str,
r->method_s,
http->request->my_addr.NtoA(myaddr,MAX_IPSTRLEN),
http->request->my_addr.GetPort());
if ((sz<=0) || (sz>=MAX_REDIRECTOR_REQUEST_STRLEN)) {
if (sz<=0) {
status = HTTP_INTERNAL_SERVER_ERROR;
debugs(61, DBG_CRITICAL, "ERROR: Gateway Failure. Can not build
request to be passed to srewriter. Request ABORTED.");
} else {
status = HTTP_REQUEST_URI_TOO_LARGE;
debugs(61, DBG_CRITICAL, "ERROR: Gateway Failure. Request passed to
srewriter exceeds MAX_REDIRECTOR_REQUEST_STRLEN (" <<
MAX_REDIRECTOR_REQUEST_STRLEN << "). Request ABORTED.");
}
clientStreamNode *node = (clientStreamNode
*)http->client_stream.tail->prev->data;
clientReplyContext *repContext = dynamic_cast<clientReplyContext
*>(node->data.getRaw());
assert (repContext);
Ip::Address tmpnoaddr;
tmpnoaddr.SetNoAddr();
repContext->setReplyToError(ERR_GATEWAY_FAILURE, status,
http->request->method, NULL,
http->getConn() != NULL &&
http->getConn()->clientConnection != NULL ?
http->getConn()->clientConnection->remote :
tmpnoaddr,
http->request,
NULL,
#if USE_AUTH
http->getConn() != NULL &&
http->getConn()->auth_user_request != NULL ?
http->getConn()->auth_user_request :
http->request->auth_user_request);
#else
NULL);
#endif
node = (clientStreamNode *)http->client_stream.tail->data;
clientStreamRead(node, http, node->readBuffer);
return;
}
debugs(61,6, HERE << "sending '" << buf << "' to the helper");
helperSubmit(srewriters, buf, srewriteHandleReply, r);
}
static void
srewriteRegisterWithCacheManager(void)
{
Mgr::RegisterAction("srewriter", "URL Srewriter Stats", srewriterStats, 0,
1);
}
void
srewriteInit(void)
{
static int init = 0;
srewriteRegisterWithCacheManager();
if (!Config.Program.srewrite)
return;
if (srewriters == NULL)
srewriters = new helper("srewriter");
srewriters->cmdline = Config.Program.srewrite;
srewriters->childs.updateLimits(Config.srewriteChildren);
srewriters->ipc_type = IPC_STREAM;
helperOpenServers(srewriters);
if (!init) {
init = 1;
CBDATA_INIT_TYPE(srewriterStateData);
}
}
void
srewriterShutdown(void)
{
if (!srewriters)
return;
helperShutdown(srewriters);
if (!shutting_down)
return;
delete srewriters;
srewriters = NULL;
}
