Re: [squid-users] Profiling patch redo, need help.
fre 2006-09-01 klockan 15:06 -0700 skrev Pranav Desai: > On 9/1/06, Henrik Nordstrom <[EMAIL PROTECTED]> wrote: > > fre 2006-09-01 klockan 10:20 -0700 skrev Pranav Desai: > > > > > For client and server timers request_t seems to work fine, but for the > > > disk In/Out timers, I can get to request_t through > > > storeIOState->e->mem_obj->request_t. Is this reasonable ? > > > > Maybe. But I don't think there is a guarantee there is a > > mem_obj->request on disk I/O. > > > > Ok. Is there any other place this info can be stored ? The natural place for per object disk I/O counters is the sio. There is no close relation sio<->HTTP request as they live independent of each other. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: [squid-users] Profiling patch redo, need help.
On 9/1/06, Henrik Nordstrom <[EMAIL PROTECTED]> wrote: fre 2006-09-01 klockan 10:20 -0700 skrev Pranav Desai: > For client and server timers request_t seems to work fine, but for the > disk In/Out timers, I can get to request_t through > storeIOState->e->mem_obj->request_t. Is this reasonable ? Maybe. But I don't think there is a guarantee there is a mem_obj->request on disk I/O. Ok. Is there any other place this info can be stored ? Thanks -- Pranav Regards Henrik -- -- http://pd.dnsalias.org
Re: [squid-users] Profiling patch redo, need help.
fre 2006-09-01 klockan 10:20 -0700 skrev Pranav Desai: > For client and server timers request_t seems to work fine, but for the > disk In/Out timers, I can get to request_t through > storeIOState->e->mem_obj->request_t. Is this reasonable ? Maybe. But I don't think there is a guarantee there is a mem_obj->request on disk I/O. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: [squid-users] Profiling patch redo, need help.
On 8/31/06, Henrik Nordstrom <[EMAIL PROTECTED]> wrote: tor 2006-08-31 klockan 15:06 -0700 skrev Pranav Desai: > I have done the other changes you suggested. I will send the patch > after some testing. I had one question though, I am not able to find > any struct that has a lifetime of an entire request. clientHttpRequest is the full lifetime of an entire request. There isn't a request until it have have been read from the connection. request_t is almost the full lifetime, and is what is used all over. It's a complete request with request headers. A connection can carry multiple requests, or even none. We have: 1. Connection accepted: ConnStateData (client_side only) 2. Request headers read: clientHttpRequest (client_side only) and request_t (core request processing) Adding timers to request_t is OK. Only odd thing to remember is that in http.c you need to use ->orig_request For client and server timers request_t seems to work fine, but for the disk In/Out timers, I can get to request_t through storeIOState->e->mem_obj->request_t. Is this reasonable ? Thanks. -- Pranav Regards Henrik -- -- http://pd.dnsalias.org
Re: [squid-users] Profiling patch redo, need help.
tor 2006-08-31 klockan 15:06 -0700 skrev Pranav Desai: > I have done the other changes you suggested. I will send the patch > after some testing. I had one question though, I am not able to find > any struct that has a lifetime of an entire request. clientHttpRequest is the full lifetime of an entire request. There isn't a request until it have have been read from the connection. request_t is almost the full lifetime, and is what is used all over. It's a complete request with request headers. A connection can carry multiple requests, or even none. We have: 1. Connection accepted: ConnStateData (client_side only) 2. Request headers read: clientHttpRequest (client_side only) and request_t (core request processing) Adding timers to request_t is OK. Only odd thing to remember is that in http.c you need to use ->orig_request Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: [squid-users] Profiling patch redo, need help.
On 8/30/06, Adrian Chadd <[EMAIL PROTECTED]> wrote: > What about the others, client and serverIn/Out ? are those the right ones ? > Also, are the location of the profile points correct, since that would > be important for getting accurate info. Hm, it should be. You may wish to use clientHttpRequest to measure the performance of individual http requests as in -theory- (not in practice yet!) squid can parse pipelined requests from clients and so more than one request per client could be pending at any time. I dont think I can use clientHttpRequest to store clientIn/Out since this is not allocated until parseHttpRequest() is called. The clientIn measurement starts in httpAccept(), where we dont have any reference to clientHttpRequest. I have done the other changes you suggested. I will send the patch after some testing. I had one question though, I am not able to find any struct that has a lifetime of an entire request. This is making it difficult to corelate the measurements. E.g. if I want to find out how much time elapsed between the client request came in to server request was sent out i.e. difference between clientIn and serverOut. I cant find that since they are in different structs. I know I can parse the logs and get that data, but if there was a easy way to do it internally, it would be nice. > Sounds good ... this should work out perfectly, I think. Good luck! Adrian -- -- http://pd.dnsalias.org
Re: [squid-users] Profiling patch redo, need help.
On 8/30/06, Adrian Chadd <[EMAIL PROTECTED]> wrote:
Try to keep this just onto squid-dev.
sure thing.
On Wed, Aug 30, 2006, Pranav Desai wrote:
> Hello,
>
> For this patch I have to use the following structures to store the
> timestamps.
> ConnStateData (for clientIn/Out)
> HttpStateData (for serverIn/Out)
> StoreEntry (for diskIn/Out)
>
> With this is becomes kind of difficult to present the result in one
> place. Do you think its possible to use StoreEntry struct for
> everything, since it seems like the all the above structs use
> StoreEntry.
Rightio. The main reason to not increase the size of StoreEntry
is that it'll blow out the memory footprint of your cache.
You could easily double the size of StoreEntry.
So I'd suggest using storeIOState for diskIn/Out. Its nice and
ephemeral. :)
What about the others, client and serverIn/Out ? are those the right ones ?
Also, are the location of the profile points correct, since that would
be important for getting accurate info.
Also: please consider using access/store.log - or create a new
logfile using the logfile.c routines - as cache.log is written
in a simple but potentially easily-bottlenecked method.
The latest snapshots of Squid-2.6 use an external process to write
access/store logs (ie, logs written through the logfile.c
code) and this'll write out very quickly.
A profiling log wouldn't be a bad idea! You'd do it this way:
Global:
Logfile *f;
Startup:
f = logfileOpen("/tmp/profile.log", 1, 1);
To write profiling logs:
logfileLineStart(f);
logfilePrintf(f, "stuff, to print, printf style", printf-style-arguments);
logfilePrintf(f, "and the rest of the line..\n", more-printf-arguments);
logfileLineEnd(f);
.. then during shutdown:
logfileClose(f);
Sounds good ... this should work out perfectly, I think.
-- Pranav
--
--
http://pd.dnsalias.org
Re: [squid-users] Profiling patch redo, need help.
> What about the others, client and serverIn/Out ? are those the right ones ? > Also, are the location of the profile points correct, since that would > be important for getting accurate info. Hm, it should be. You may wish to use clientHttpRequest to measure the performance of individual http requests as in -theory- (not in practice yet!) squid can parse pipelined requests from clients and so more than one request per client could be pending at any time. > Sounds good ... this should work out perfectly, I think. Good luck! Adrian
Re: [squid-users] Profiling patch redo, need help.
Try to keep this just onto squid-dev.
On Wed, Aug 30, 2006, Pranav Desai wrote:
> Hello,
>
> For this patch I have to use the following structures to store the
> timestamps.
> ConnStateData (for clientIn/Out)
> HttpStateData (for serverIn/Out)
> StoreEntry (for diskIn/Out)
>
> With this is becomes kind of difficult to present the result in one
> place. Do you think its possible to use StoreEntry struct for
> everything, since it seems like the all the above structs use
> StoreEntry.
Rightio. The main reason to not increase the size of StoreEntry
is that it'll blow out the memory footprint of your cache.
You could easily double the size of StoreEntry.
So I'd suggest using storeIOState for diskIn/Out. Its nice and
ephemeral. :)
Also: please consider using access/store.log - or create a new
logfile using the logfile.c routines - as cache.log is written
in a simple but potentially easily-bottlenecked method.
The latest snapshots of Squid-2.6 use an external process to write
access/store logs (ie, logs written through the logfile.c
code) and this'll write out very quickly.
A profiling log wouldn't be a bad idea! You'd do it this way:
Global:
Logfile *f;
Startup:
f = logfileOpen("/tmp/profile.log", 1, 1);
To write profiling logs:
logfileLineStart(f);
logfilePrintf(f, "stuff, to print, printf style", printf-style-arguments);
logfilePrintf(f, "and the rest of the line..\n", more-printf-arguments);
logfileLineEnd(f);
.. then during shutdown:
logfileClose(f);
Re: [squid-users] Profiling patch redo, need help.
Hello, For this patch I have to use the following structures to store the timestamps. ConnStateData (for clientIn/Out) HttpStateData (for serverIn/Out) StoreEntry (for diskIn/Out) With this is becomes kind of difficult to present the result in one place. Do you think its possible to use StoreEntry struct for everything, since it seems like the all the above structs use StoreEntry. Let me know if you have any other suggestions as well. Thanks for your time. -- Pranav Some results for the patch: from wget -r -l 1 www.google.com 2006/08/30 15:08:09| PROFILING: clientIn http://www.google.com/ 1156975689.70644/63/105 2006/08/30 15:08:09| PROFILING: serverOut / 1156975689.159148/129/168 2006/08/30 15:08:09| PROFILING: serverIn / 1156975689.159355/103444/107351 2006/08/30 15:08:09| PROFILING: clientOut / 1156975689.262965/194/3988 2006/08/30 15:08:09| PROFILING: clientIn http://www.google.com/robots.txt 1156975689.268151/32/61 2006/08/30 15:08:09| PROFILING: serverOut /robots.txt 1156975689.357594/161/227 2006/08/30 15:08:09| PROFILING: serverIn /robots.txt 1156975689.357879/110398/111290 2006/08/30 15:08:09| PROFILING: clientOut /robots.txt 1156975689.468400/167/849 2006/08/30 15:08:11| PROFILING: clientIn http://www.google.com/intl/en/images/logo.gif 1156975691.101383/43/81 2006/08/30 15:08:11| PROFILING: serverOut /intl/en/images/logo.gif 1156975691.203600/94/131 2006/08/30 15:08:11| PROFILING: serverIn /intl/en/images/logo.gif 1156975691.203767/109384/247216 2006/08/30 15:08:11| PROFILING: diskOut http://www.google.com/intl/en/images/logo.gif 1156975691.317039/276/133993 2006/08/30 15:08:11| PROFILING: clientOut /intl/en/images/logo.gif 1156975691.313285/179/137957 2006/08/30 15:08:11| PROFILING: clientIn http://www.google.com/imghp?hl=en&ie=UTF-8&tab=wi 1156975691.451883/31/60 2006/08/30 15:08:11| PROFILING: serverOut /imghp?hl=en&ie=UTF-8&tab=wi 1156975691.452076/75/119 2006/08/30 15:08:11| PROFILING: serverIn /imghp?hl=en&ie=UTF-8&tab=wi 1156975691.452212/393940/398887 2006/08/30 15:08:11| PROFILING: clientOut /imghp?hl=en&ie=UTF-8&tab=wi 1156975691.846269/254/4921 2006/08/30 15:08:11| PROFILING: clientIn http://www.google.com/maphp?hl=en&ie=UTF-8&tab=wl 1156975691.851874/29/58 2006/08/30 15:08:11| PROFILING: serverOut /maphp?hl=en&ie=UTF-8&tab=wl 1156975691.970916/95/135 2006/08/30 15:08:12| PROFILING: serverIn /maphp?hl=en&ie=UTF-8&tab=wl 1156975691.971091/137047/537778 2006/08/30 15:08:12| PROFILING: clientOut /maphp?hl=en&ie=UTF-8&tab=wl 1156975692.108259/161/400726 2006/08/30 15:08:12| PROFILING: clientIn http://www.google.com/intl/en/options/ 1156975692.509833/30/59 2006/08/30 15:08:12| PROFILING: serverOut /intl/en/options/ 1156975692.748149/93/130 2006/08/30 15:08:13| PROFILING: serverIn /intl/en/options/ 1156975692.748318/115016/355249 2006/08/30 15:08:13| PROFILING: clientOut /intl/en/options/ 1156975692.863462/195/240267 2006/08/30 15:08:13| PROFILING: clientIn http://www.google.com/language_tools?hl=en 1156975693.104599/36/67 2006/08/30 15:08:13| PROFILING: serverOut /language_tools?hl=en 1156975693.233595/100/137 2006/08/30 15:08:14| PROFILING: serverIn /language_tools?hl=en 1156975693.233769/316264/1075800 2006/08/30 15:08:14| PROFILING: clientOut /language_tools?hl=en 1156975693.550152/162/759534 2006/08/30 15:08:14| PROFILING: clientIn http://www.google.com/intl/en/ads/ 1156975694.666853/41/79 2006/08/30 15:08:14| PROFILING: serverOut /intl/en/ads/ 1156975694.755357/91/128 2006/08/30 15:08:14| PROFILING: serverIn /intl/en/ads/ 1156975694.755520/234571/239791 2006/08/30 15:08:14| PROFILING: clientOut /intl/en/ads/ 1156975694.990272/187/5136 2006/08/30 15:08:14| PROFILING: clientIn http://www.google.com/services/ 1156975694.996256/29/58 2006/08/30 15:08:15| PROFILING: serverOut /services/ 1156975695.98481/94/132 2006/08/30 15:08:15| PROFILING: serverIn /services/ 1156975695.98681/98387/438090 2006/08/30 15:08:15| PROFILING: clientOut /services/ 1156975695.197195/163/339670 2006/08/30 15:08:15| PROFILING: clientIn http://www.google.com/intl/en/about.html 1156975695.537512/37/67 2006/08/30 15:08:15| PROFILING: serverOut /intl/en/about.html 1156975695.650173/97/130 2006/08/30 15:08:15| PROFILING: serverIn /intl/en/about.html 1156975695.650343/119268/222970 2006/08/30 15:08:15| PROFILING: clientOut /intl/en/about.html 1156975695.769738/171/103661 On 8/30/06, Pranav Desai <[EMAIL PROTECTED]> wrote: On 8/29/06, Pranav Desai <[EMAIL PROTECTED]> wrote: > On 8/29/06, Adrian Chadd <[EMAIL PROTECTED]> wrote: > > On Mon, Aug 28, 2006, Pranav Desai wrote: > > > > > I think I did join, but I never received the acknowledgement mail. > > > Maybe I will try it again. > > > > I've manually added you now. > > > > Great ! thanks. > > > >You can just include the patch in the body of your email rather than as > > > >an attachment. Or, if its more than a couple hundred lines long, just > > > >put it on a website somewhere. > > > > > > > > > > > Here is the patch > > > > > > http://pd.dnsalias.org/downloa
Re: [squid-users] Profiling patch redo, need help.
On 8/29/06, Pranav Desai <[EMAIL PROTECTED]> wrote: On 8/29/06, Adrian Chadd <[EMAIL PROTECTED]> wrote: > On Mon, Aug 28, 2006, Pranav Desai wrote: > > > I think I did join, but I never received the acknowledgement mail. > > Maybe I will try it again. > > I've manually added you now. > Great ! thanks. > > >You can just include the patch in the body of your email rather than as > > >an attachment. Or, if its more than a couple hundred lines long, just > > >put it on a website somewhere. > > > > > > > > Here is the patch > > > > http://pd.dnsalias.org/download/profiling_patch > Please download the updated patch. I had missed some files. cd squid-2.6.STABLE3 patch -p1 < profiling_patch export CFLAGS="-DPROFILE_OBJECTS" ./configure make -- Pranav > Ok! Have you run it yet? What kind of results are you seeing? > Yes, I have run some really basic test, few requests, it seems correct since the timing at each step seems reasonable and also the duration of the entire request processing matches the time shown in access.log. > > > Adrian > > -- -- http://pd.dnsalias.org -- -- http://pd.dnsalias.org
Re: [squid-users] Profiling patch redo, need help.
On 8/29/06, Adrian Chadd <[EMAIL PROTECTED]> wrote: On Mon, Aug 28, 2006, Pranav Desai wrote: > I think I did join, but I never received the acknowledgement mail. > Maybe I will try it again. I've manually added you now. Great ! thanks. > >You can just include the patch in the body of your email rather than as > >an attachment. Or, if its more than a couple hundred lines long, just > >put it on a website somewhere. > > > > > Here is the patch > > http://pd.dnsalias.org/download/profiling_patch Ok! Have you run it yet? What kind of results are you seeing? Yes, I have run some really basic test, few requests, it seems correct since the timing at each step seems reasonable and also the duration of the entire request processing matches the time shown in access.log. Adrian -- -- http://pd.dnsalias.org
Re: [squid-users] Profiling patch redo, need help.
On Mon, Aug 28, 2006, Pranav Desai wrote: > I think I did join, but I never received the acknowledgement mail. > Maybe I will try it again. I've manually added you now. > >You can just include the patch in the body of your email rather than as > >an attachment. Or, if its more than a couple hundred lines long, just > >put it on a website somewhere. > > > > > Here is the patch > > http://pd.dnsalias.org/download/profiling_patch Ok! Have you run it yet? What kind of results are you seeing? Adrian
