Re: how to really bang on a script?
On Sat, 28 Oct 2000, Christopher L. Everett wrote: So, I apologize for not describing my problem clearly in the first place. And again, my questions are: How would I go about proving to myself that my script does what I designed it to do? Has anyone else dealt with a similar problem, and how did they go about doing it? If I solve it for myself, would anyone else find the solution useful, and how would I make it more useful to them? Basically you've got exactly the right idea, and it *has* been done before - I recall vaguely in the back of my head someone mentioning doing almost exactly the same thing (changing server logs into a benchmark tool) at ApacheCon, only I can't for the life of me remember who it was. On the plus side though, provided you've got no POST parameters, its a pretty trivial script. The hard part is getting a scalable engine to execute all those requests. Really you have to do that in C I think - and you should probably just look towards the source of ab. FWIW, I think it was Theo Schlossnagel (mod_backhand guy) talking about the utility, in fact I'm almost certain. Why don't you drop him a line. -- Matt/ /||** Director and CTO ** //||** AxKit.com Ltd ** ** XML Application Serving ** // ||** http://axkit.org ** ** XSLT, XPathScript, XSP ** // \\| // ** Personal Web Site: http://sergeant.org/ ** \\// //\\ // \\
Re: how to really bang on a script?
On Sat, 28 Oct 2000, Matt Sergeant wrote:
exactly the same thing (changing server logs into a benchmark tool) at
ApacheCon, only I can't for the life of me remember who it was.
Theo, during the mod_backhand talk, or at lunch just before, I can't
remember.
MBM
--
Matthew Byng-Maddick Home: [EMAIL PROTECTED] +44 20 8981 8633 (Home)
http://colondot.net/ Work: [EMAIL PROTECTED] +44 7956 613942 (Mobile)
perl -e '$_="Oyvv bsswjfw Thtm mefmfw2\n";while(m([^\n])){$_=$'"'"';$a=$;
$a=($a=~m(^\s)?$a:pack "c",unpack("c",$a)-5+($i++%5));print $a}print"\n";'
Re: ApacheCon report
Matt Sergeant wrote: http://modperl.sergeant.org/ApacheConRep.txt Enjoy. Thanks for that Matt, I did enjoy it - IBM's party coninciding with Suns keynote made me chukle ;-) I eventually could not make the conferance due to a nasty deadline Did Doug mention when mod_perl 2.0 would / maybe / migh possibly be ready (I know, I know that it will be ready when its ready, only asking!) Greg -- Matt/ /||** Director and CTO ** //||** AxKit.com Ltd ** ** XML Application Serving ** // ||** http://axkit.org ** ** XSLT, XPathScript, XSP ** // \\| // ** Personal Web Site: http://sergeant.org/ ** \\// //\\ // \\
Re: ApacheCon report
On Sat, 28 Oct 2000, Greg Cope wrote:
Matt Sergeant wrote:
http://modperl.sergeant.org/ApacheConRep.txt
Enjoy.
Thanks for that Matt, I did enjoy it - IBM's party coninciding with Suns
I eventually could not make the conferance due to a nasty deadline
You missed a lot.
Did Doug mention when mod_perl 2.0 would / maybe / migh possibly be
ready (I know, I know that it will be ready when its ready, only
asking!)
I don't remember him mentioning anything, but it will certainly have to
wait until all the bugs in apache 2.0 are fixed. :) (hopes that rbb
doesn't read this...)
According to various asf people, apache2 is at least a month away from
being finished, and probably more...
MBM
--
Matthew Byng-Maddick Home: [EMAIL PROTECTED] +44 20 8981 8633 (Home)
http://colondot.net/ Work: [EMAIL PROTECTED] +44 7956 613942 (Mobile)
perl -e '$_="Oyvv bsswjfw Thtm mefmfw2\n";while(m([^\n])){$_=$'"'"';$a=$;
$a=($a=~m(^\s)?$a:pack "c",unpack("c",$a)-5+($i++%5));print $a}print"\n";'
Re: ApacheCon report
On Sat, 28 Oct 2000, Greg Cope wrote: Matt Sergeant wrote: http://modperl.sergeant.org/ApacheConRep.txt Enjoy. Thanks for that Matt, I did enjoy it - IBM's party coninciding with Suns keynote made me chukle ;-) I eventually could not make the conferance due to a nasty deadline Did Doug mention when mod_perl 2.0 would / maybe / migh possibly be ready (I know, I know that it will be ready when its ready, only asking!) Unfortunately I had to run (I had a beer with my name on it) before getting chance to speak to Doug again. But it also depends a *lot* on the progress of Apache 2.0, which is really holding mod_perl 2.0 up, from what I can tell. But once that's out of the way, I don't think anything is stopping mod_perl 2.0 progress - Doug seems to be able to move pretty quickly on things, as he has a good idea where he's going with the project. -- Matt/ /||** Director and CTO ** //||** AxKit.com Ltd ** ** XML Application Serving ** // ||** http://axkit.org ** ** XSLT, XPathScript, XSP ** // \\| // ** Personal Web Site: http://sergeant.org/ ** \\// //\\ // \\
Re: how to really bang on a script?
Chris, i'd bet my head a few months ago someone announced an apache::bench module, that would take a log and run it as a benchmarking secuence of HTTP requests. just get to the list archives and start searching with benchmarks and logs. CPAN is your friend, also. there are at least 2 or 3 benching perl scripts available. I bet at least one does what you need. but I may still loose my bet ... m
Re: ApacheCon report
Matt Sergeant wrote: On Sat, 28 Oct 2000, Greg Cope wrote: Matt Sergeant wrote: http://modperl.sergeant.org/ApacheConRep.txt Enjoy. Thanks for that Matt, I did enjoy it - IBM's party coninciding with Suns keynote made me chukle ;-) I eventually could not make the conferance due to a nasty deadline Did Doug mention when mod_perl 2.0 would / maybe / migh possibly be ready (I know, I know that it will be ready when its ready, only asking!) Unfortunately I had to run (I had a beer with my name on it) before getting chance to speak to Doug again. But it also depends a *lot* on the progress of Apache 2.0, which is really holding mod_perl 2.0 up, from what I can tell. But once that's out of the way, I don't think anything is stopping mod_perl 2.0 progress - Doug seems to be able to move pretty quickly on things, as he has a good idea where he's going with the project. Sounds sooner than I though - great. Doug and Ryan appear to work for the same company - Covalent - so Doug should go bend Ryans ear - I want my mod_perl 2.0 ;-) Thanks again Matt. Greg -- Matt/ /||** Director and CTO ** //||** AxKit.com Ltd ** ** XML Application Serving ** // ||** http://axkit.org ** ** XSLT, XPathScript, XSP ** // \\| // ** Personal Web Site: http://sergeant.org/ ** \\// //\\ // \\
Re: how to really bang on a script?
Hi again, On Sat, 28 Oct 2000, Christopher L. Everett wrote: OK, I confess: I've written (probably yet another) mod_perl banner exchange. Argh. [snipped impressive numbers] That's what I meant by "certain statistical properties". Interesting. Nothing to do with statistics, but interesting. So, I apologize for not describing my problem clearly No need for that. I need to know that the 500 sites that hosted our banner ads are getting the 50K banners that we promised them The only way to *know* that is to get information from the sites. If I wanted to be sure, I'd connect to them and look in their logs. You probaly already know that people cheat a lot with this kind of thing. Can't understand why they'd do that. and the 30K banners that we sold, we really did serve. Is there a reason you're only selling - no, don't answer that. Also, I want to know that the banners my logs say the script sent are really the ones people saw on their browsers. There you go again. The only way to know is to go and look. But you might try something along the lines of tcpdump on your firewall box, so that you could at least remove a whole slew of uncertainties from the equation. You could pull your magic numbers out from the packets and see from the packet headers if they were addressed to where they were suposed to be addressed to. But without going further afield I think that's about the best you will be able to do. And don't write off ab - remember, you saw it here first... 73, Ged.
Re: Apache::GzipChain
Hi there, On Sat, 28 Oct 2000, Jerrad Pierce wrote: Is anybody using GzipChain? IIRC, Josh said he was. He didn't complain about it. Raved, in fact. Is there some known means of verifying that it is in fact working properly? LWP? 73, Ged.
Re: how to really bang on a script?
"Christopher L. Everett" wrote:
Hello All:
I've written some mod_perl scripts that need testing
over a million hits or so before I deploy it. I need
to prove to myself and my marketing guy that my script
has certain statistical properties, not the least of
which is the question of whether my activity logs match
what actually happened. Also, there's concurrency
issues to make sure I've got right.
snip
sorry, but i fail to see why all the trickery is needed. i assume that
you want to check the content against what is expected, but the banners
are rotating based on some formula. if you know the formula before hand,
then you know the expected distribution for the banners served. why not
use something from libwww package to make the requests, md5 the returned
banner relavant data, return a report which gives the counts for each unique md5.
so in perl pseudocode (untested, no error checking, steps skipped):
for $testnum (0..$number_to_test) {
$request_url=@bannerurls[rand(@bannerurls)];
$req = HTTP::Request-new(GET = $request_url);
$res=$ua-request($req);
if($res-is_success){
$dig=md5($res-content);
$md5{$dig}++;
}
$md5{__ERROR__}++;
}
}
foreach $dig (keys %md5){
print "the banner with digest=$dig returned $md5{$dig} hits
for",($md5{$dig}/$number_to_test)*100),"% of total\n";
}
the report should print the distribution requested. not sure if this can be done
using ab or the bench
i believe Stas was working on, but libwww is easy to use. you can even distribute
this test to a few
of your friends and have them bang on your system from a variety of different places
to test your
system in a more realistic environment. then write a program to scour the logs for
the test period,
and produce the same report from the logs. they should match almost exactly. only
differences in
incompleted log requests etc..
1) Is there a more elegant way of solving my problem?
2) Has this been done before?
2a) If so, is the source for that available?
2b) If not, is a tool like this useful for anyone else,
so that I should build it better than I would a once-off?
What would make it more useful?
Thanks in advance for your help.
--Christopher Everett
[EMAIL PROTECTED]
641-472-4178
--
___cliff [EMAIL PROTECTED]http://www.genwax.com/
Re: Apache::GzipChain
On Sat, 28 Oct 2000, G.W. Haywood wrote: Hi there, On Sat, 28 Oct 2000, Jerrad Pierce wrote: Is anybody using GzipChain? IIRC, Josh said he was. He didn't complain about it. Raved, in fact. Is there some known means of verifying that it is in fact working properly? LWP? better to use your logs... LWP won't trigger it... if you download a 100k page when you look at "view page info" or save it to disk, but your access log shows 15k, you know it's doing it's job... (hhmm, seems I was using Apache::Gzip until I got my ADSL back ... but at that time, it was a non-trivial exercise, and compressing _everything_ (including PHP scripts etc...) required using LWP internally, which worked even better for checking functionality, because you had two log entries, the raw one from localhost, and the one compressed one from the remote agent, with "appropriate" variances in their sizes :-) 73, Ged. -- [EMAIL PROTECTED] | Don't go around saying the world owes you a living; http://BareMetal.com/ | the world owes you nothing; it was here first. web hosting since '95 | - Mark Twain
Re: how to really bang on a script?
On Sat, 28 Oct 2000, martin langhoff wrote: Chris, i'd bet my head a few months ago someone announced an apache::bench module, that would take a log and run it as a benchmarking secuence of HTTP requests. just get to the list archives and start searching with I wrote a simple perl script (that forks multiple childredn and uses IPCs to get multiple threads banging on your box) that runs from a parsed log... but it was more to test functionality than as a benchmarking tool. It _should_ still be floating around here... benchmarks and logs. CPAN is your friend, also. there are at least 2 or 3 benching perl scripts available. I bet at least one does what you need. but I may still loose my bet ... m -- [EMAIL PROTECTED] | Don't go around saying the world owes you a living; http://BareMetal.com/ | the world owes you nothing; it was here first. web hosting since '95 | - Mark Twain
Re: Connection Pooling / TP Monitor
- Original Message - From: "Matt Sergeant" [EMAIL PROTECTED] . To redirect incoming url's that require database work to mod_perl 'heavy' servers? Just like a smarter and more dynamic mod_rewrite? Yes? Yes basically, except its not a redirect. mod_backhand can use keep-alives to ensure that it never has to recreate a new connection to the heavy backend servers, unlike mod_rewrite or mod_proxy. And it can do it in a smart way so that remote connections don't use keepalives (because they are evil for mod_perl servers - see the mod_perl guide), but backhand connections do. Very very cool technology. Is there any way to tie proxy requests mapped by mod_rewrite to a balanced set of servers through mod_backhand (or anything similar)?Also, can mod_backhand (or any alternative) work with non-apache back end servers?I'm really looking for a way to let mod_rewrite do the first cut at deciding where (or whether) to send a request, but then be able to send to a load balanced, fail over set, preferably without having to interpose another physical proxy. Les Mikesell [EMAIL PROTECTED]
