Re: Chaining Perl and C handlers
Ged, Thanks for the input-- here are my answers below... The new parameter (which in the example was moe and not joe, my mistake) will be a "webid", a unique identifier for each website that is parsed out of the physical location of the file. For example, www.joe-honda-dealer.com/inventory.jsp corresponds to something like: /web/active/honda/joedeal/html/inventory.jsp where joedeal is the webid. Instead of having a bunch of identical inventory pages, however, we're just going to have one JSP on the Weblogic server that accepts a webid parameter which is responsible for how the page gets built. Let me know if you need further clarification on this. My main issue is that I don't know how to add parameters to the POST and GET data. Again, thanks for the help!! Andrew Chen Software Engineer, Architecture [EMAIL PROTECTED] 206-219-8445 The Cobalt Group, Inc. (www.cobaltgroup.com) 2200 First Avenue South Seattle, WA 98134 On Wed, 15 Nov 2000, G.W. Haywood wrote: Hi there, On Tue, 14 Nov 2000, Andrew Chen wrote: 1. Request comes in for t.jsp?test=1joe=2 2. Handler steps in and adds another parameter, t.jsp?test=1joe=2moe=3 3. Request is then handled by BEA-Apache bridge so that it is forwarded to Weblogic 4. JSP runs on Weblogic, and the new parameter (joe) moe? can be referenced in the JSP. I'm not familiar with this bridge, but I don't see why this needs to be complicated. What determines the new parameter? After the handler adds the new parameter, could you get it to make another request which then gets caught by mod_rewrite and/or mod_proxy for example? 73, Ged.
Re: Chaining Perl and C handlers
Ged, What we are doing is having requests for files of extension *.jsp be passed from Apache to a BEA Weblogic box to be handled. However, we want to add an extra parameter into the GET/POST data before the request is passed. Thus, these are the steps: 1. Request comes in for t.jsp?test=1joe=2 2. Handler steps in and adds another parameter, t.jsp?test=1joe=2moe=3 3. Request is then handled by BEA-Apache bridge so that it is forwarded to Weblogic 4. JSP runs on Weblogic, and the new parameter (joe) can be referenced in the JSP. In order to do this, I was thinking that I could either chain the handlers (have the custom handler alter the data in the request and then pass it onto the BEA-Apache bridge) or I could just put the custom handler in an earlier stage and alter the request then. I'm just not sure how to do it. Let me know if you need any more clarification! Thanks a lot! Andrew Chen Software Engineer, Architecture [EMAIL PROTECTED] 206-219-8445 The Cobalt Group, Inc. (www.cobaltgroup.com) 2200 First Avenue South Seattle, WA 98134 On Tue, 14 Nov 2000, G.W. Haywood wrote: Hi ther, On Mon, 13 Nov 2000, Andrew Chen wrote: Is there any way to maybe sneak into the handler cycle before the BEA-Apache handler and alter the GET/POST data, thus accomplishing the same thing? Let me know if you need any clarification. I need clarification. :) 73, Ged.
Re: Chaining Perl and C handlers
Thanks for the reply! Is there any way to maybe sneak into the handler cycle before the BEA-Apache handler and alter the GET/POST data, thus accomplishing the same thing? Let me know if you need any clarification. (any replies should go to my e-mail address, as I'm not subscribed to the mod_perl list) Andrew Chen Software Engineer, Architecture [EMAIL PROTECTED] 206-219-8445 The Cobalt Group, Inc. (www.cobaltgroup.com) 2200 First Avenue South Seattle, WA 98134 On Tue, 7 Nov 2000, G.W. Haywood wrote: Hi there, On Mon, 6 Nov 2000, Andrew Chen wrote: According to my trusty Eagle book, I can chain Perl handlers (by using $r-pushhandlers(), but can I chain a C handler after a Perl handler?? Not until Apache 2.0, Eagle Book p175, "Chaining Content Handlers". A C handler either handles the entire request or returs declined. 73, Ged.
Chaining Perl and C handlers
All, The project I'm working on requres me to do the following: 1. GET/POST Request comes in 2. Request gets handled by a mod_perl handler, which modifies and adds to the GET/POST variables. 3. Request is then sent to a C handler (BEA WLS-Apache bridge) 4. Request is then handled by a JSP, which should be able to read the changes that were added in step 2. According to my trusty Eagle book, I can chain Perl handlers (by using $r-pushhandlers(), but can I chain a C handler after a Perl handler?? And yes, I know this is kind of bizarre. Thanks in advance! I'm not on the mod_perl list, so please cc: me. Andrew Chen Software Engineer, Architecture [EMAIL PROTECTED] 206-219-8445 The Cobalt Group, Inc. (www.cobaltgroup.com) 2200 First Avenue South Seattle, WA 98134
Apache::PerlRun::Flush
Hello all, There was some discussion about the Apache::PerlRun::Flush handler before, but here's a quick refresher: the handler was written in order to be able to run dirty code on PerlRun with PerlRunOnce Off by going through and cleaning up the memory space between each request. It's been working okay, and we're putting it through QA right now. Some people have e-mailed and asked about open sourcing of the handler, and all I can say is: It will be done, I'm just not sure when. This will be the first piece of software open sourced here at Cobalt, and it just needs to go through the right channels. By the way, is having "Apache" in the name of the package a problem? To the technical issue: One of the only problems about the handler right now is that there seems to be a strange memory leak. If I have Apache running and serving requests, if I watch the httpds on top I see them use more and more memory until it starts taxing the server. Everything seems to be working fine (variables seem to be getting flushed, etc.) but this is still happening. Does anyone have any ideas or any way to track down what variables are using up the memory? When I have Apache::Status loaded up, the Apache::ROOT* temporary PerlRun packages that get created seem to be occupying a nonzero amount of space. Is this a PerlRun bug? Please let me know if you have any idas-- I'm not on the list so please cc: me. Andrew Chen Intern, Architecture [EMAIL PROTECTED] 206-219-8445 The Cobalt Group, Inc.
Re: Flushing handler getting Open Sourced...
Yeah, that sounds about right... It's not at either extreme-- it doesn't keep EVERYTHING, and it doesn't kill the Apache child to clear the memory. I'm sure the exact directives can be thought out later. So who do I get in touch with in order for this stuff to get looked at for inclusion into mod_perl? Andrew Chen Intern, Architecture [EMAIL PROTECTED] 206-219-8445 The Cobalt Group, Inc. On Tue, 8 Aug 2000, Ken Williams wrote: [EMAIL PROTECTED] (Andrew Chen) wrote: Hello all, I was involved in a discussion a few weeks ago regarding how Cobalt Group (www.cobaltgroup.com) could get a performance boost by running our very very dirty modules under PerlRun with PerlRunOnce Off and then to have a handler that cleaned up between requests. Anyway, with a lot of help from the mod_perl community (thanks guys!) the handler was written, and it is currently going through QA to push this out to all of our websites (many thousands). Holy ambition! Additionally, we are going to release this back into the community for those that, like us, are unlucky enough to have dirty code that would be too expensive to rewrite. The question is now: How should we do it? 1) Should it be integrated into the mod_perl package? 2) Should it be a seperate package in CPAN? 3) etc., etc. It's quite possible that some of your functionality should get integrated into PerlRun itself. It sounds like an intermediate setting between PerlRunOnce Off and PerlRunOnce On. What about a setting like PerlSetVar PerlRunMode Once(equivalent to PerlRunOnce On) PerlSetVar PerlRunMode Cleanup (adds your cleanup stuff) PerlSetVar PerlRunMode Trust (equivalent to PerlRunOnce Off) The 'Trust' name is pretty stupid, couldn't think of anything better right now. Anyway, would something like this make sense? Or perhaps a "PerlRunCleanup" directive?
Flushing handler getting Open Sourced...
Hello all, I was involved in a discussion a few weeks ago regarding how Cobalt Group (www.cobaltgroup.com) could get a performance boost by running our very very dirty modules under PerlRun with PerlRunOnce Off and then to have a handler that cleaned up between requests. Anyway, with a lot of help from the mod_perl community (thanks guys!) the handler was written, and it is currently going through QA to push this out to all of our websites (many thousands). Additionally, we are going to release this back into the community for those that, like us, are unlucky enough to have dirty code that would be too expensive to rewrite. The question is now: How should we do it? 1) Should it be integrated into the mod_perl package? 2) Should it be a seperate package in CPAN? 3) etc., etc. Anyway, if anyone has an idea of what the best way to go about this would be, that would be great. Thanks! Andrew Chen Intern, Architecture [EMAIL PROTECTED] 206-219-8445 The Cobalt Group, Inc.
Re: mod_perl success!
It's actually: PerlSetVar PerlRunOnce On (that's without the handler, of course. With it you want that off) Andrew Chen Intern, Architecture [EMAIL PROTECTED] 206-219-8445 The Cobalt Group, Inc. On Tue, 1 Aug 2000, Pramod Sokke wrote: Where did you set your PerlRunOnce option? I tried setting it in httpd.conf like this: Alias /cgi-perl/ /usr/ns-home/cgi-bin/ Location /cgi-perl/ #AllowOverride None SetHandler perl-script PerlHandler Apache::PerlRun Options +ExecCGI allow from all PerlSendHeader On PerlRunOnce On /Location and I got this error: Invalid command 'PerlRunOnce', perhaps mis-spelled or defined by a module not included in the server configuration. -Pramod At 04:25 PM 7/25/00 -0700, Andrew Chen wrote: On Tue, 25 Jul 2000, Pramod Sokke wrote: Andrew, From your discussion on the forum, I recall you mentioning the differences between PerlRunOnce being 'On' and 'Off'. Since our code is extremely shabby and huge, will I be able to use it with 'Off'? If I don't, will I be compromising on the speed? Thanks again for the great detailed write-up! -Pramod If PerlRunOnce is On, then you can use your code without any changes, even if it is completely horrible. PerlRunOnce Off allows for bigger speed increases, but then you have to write the handler. If you just want a really quick speed improvement, stick it on PerlRun with PerlRunOnce On and see if that's sufficient. If you feel like writing the handler (so that you can get a really big speed boost) then you can run it with PerlRunOnce Off. Andrew
Re: mod_perl success!
Pramod:
We are actually going to be generalizing it and releasing it (maybe) but
we need to get approval first, so can't just send the code :) It'll
probably happen in a few weeks though.
So here's what you have to do:
- Clean up the systems-level modules (like database stuff, base libraries)
and packages that are used a lot (stuff like date conversions and other
random operations) by making sure they pass "perl -w" and "use strict"
- Install mod_perl into Apache
- Make Apache handle CGIs through PerlRun with PerlRunOnce Off
- Put the clean packages that you want to keep in memory in a startup.pl
file that is PerlRequire'd by the httpd.conf file
- Write the handler (more details below)
- After the PerlRequire statement, install the handler as a
PerlCleanupHandler
That should do it.
Here are the details about the cleanup handler in pseudocode:
BEGIN {
load the list of packages that are in memory now into a global variable
}
sub handler {
look through the list of packages in memory and see if there are entries
in the global variable
if not, then flush them out (code you can copy from PerlRun.pm in
mod_perl source)
after flushing it out, then clear the entry in %INC
}
The BEGIN {} makes it so that you NEED to load this handler after the
startup.pl stuff because otherwise the handler will flush the clean
packages you have listed in the startup.pl file.
Also, we actually wrote a database package that acts exactly like
Apache::DBI (but has some customized features) so we didn't end up using
the Apachage::DBI package. However, for persistent connections this might
be something that you want to look into.
I hope this makes sense. Let me know if you have any questions. If you can
wait a few weeks (2-3) you can just grab the handler from me :) Just so
you know, after all the optimizations that we have been able to
practically deal with (cleaning systems modules, that kind of thing) we
are getting 100%-400% speed increases. Pretty nice, huh?
Andrew Chen
Intern, Architecture
[EMAIL PROTECTED]
206-219-8445
The Cobalt Group, Inc.
On Tue, 25 Jul 2000, Pramod Sokke wrote:
Hi Andrew,
I've been following your mails regarding porting your CGIs to mod_perl on
the mod_perl mailing list, but I got lost after a few mails. But I know
that you celebrated success at the end of it allanother mod_perl sucess.
Basically, the reason I'm writing to you now is because I have the exact
same challenge as you had when you started this project - migrate CGIs to
mod_perl. As with your case, our code is too dirty to use Apache::Registry,
so I have to use Apache::PerlRun. Our code base is huge and our application
is very database intense, in the sense that there are too many calls to the
database.
I would appreciate if you could tell me in as much detail as possible about
how exactly you went about doing this migration. I know this is too much
trouble for you, but you'll be helping another soul a lot :)
I'm not in a big hurry, so you can take your time to write this up. It'll
be really useful if you can give me as much details as possible. If you
already have a document that you wrote about this, that would great too.
Thanks in advance!
-Pramod
Re: PerlRun question [RESOLVED]
Perrin, Ken, Honza, and all others that helped: The problem was actually that I wasn't clearing %INC properly after flushing out all the dirty modules. After doing that everything started working properly. So anyway, I wrote an Perl script and installed it as an PerlCleanupHandler. I stole code from the PerlRun source code so that, instead of just flushing it's immediate variables, it actually actively looks around into other packages and clears them out as well. Thus we are flushing out some of the dirty modules that were causing us problems while keeping the clean modules in memory. Basically we have the advantages of PerlRun with PerlRunOnce Off without the headache of changing the existing codebase (even though there is a little overhead with this cleanup script). The affect of this is that we have database pooling as well as a significant performance boost (between 50%-200%) without having to change a single line of code. We plan on slowly migrating all our code to add to our "keep" list, which will increase performance even more. Another mod_perl success :) Anyway, thanks a lot-- this mailing list really did help a lot. Andrew
Re: PerlRun question
Thanks for the idea. I went ahead and implemented this, and am very close. I currently have a script looks at all the variables and undefines all of them with a few exceptions (code copied from PerlRun.pm of mod_perl source). The way that the exceptions are managed is that I keep a list of the variables that I want to keep, and clear the variables that aren't on the list out. In the to-keep list, there's the obvious stuff (database and other base libraries) but also I ran a blank script and kept all the variables brought up in that blank list (Apache::*, c.). This way I'm not clearing stuff that I'm supposed to. One issue that I have at this point is that although the script runs and seems to do what it's supposed to do, it breaks stuff. If I go through these steps 1) Start httpd 2) Run flushvars.cgi (nothing flushed, because it's clean) 3) Go to a cgi that loads a bunch of modules 4) Run flushvars.cgi (a bunch of stuff is flushed-- just the modules that were loaded in step 3) 5) Go to a cgi that uses a lot of the modules that were flushed in step 4 6) It breaks I think it's trying to call a method for an object that doesn't exist anymore. My guess is that I am keeping a certain variable that keeps a list of the object in memory or something similar that I should really be flushing. Because I'm protecting it, the list doesn't get refreshed. Does anyone know the solution to this problem? Thanks for your time-- this is really turning out a lot better than I had initially hoped. Cc:: me on the reply because I am not subscribed to the mod_perl mailing list. Andrew Chen Intern, Architecture [EMAIL PROTECTED] 206-219-8445 The Cobalt Group, Inc. On Fri, 14 Jul 2000, Perrin Harkins wrote: On Fri, 14 Jul 2000, Andrew Chen wrote: You can actually do this from one place, iterating through a list of namespaces to flush. Look at the code in Apache::PerlRun::flush_namepsace. It's pretty easy. We would still have to add code to every module, correct? Even though there could be a central variable flushing module, all the other modules would still want to call it. So although we may not be including the flush_namespace code in every module, we will at least need to call them. No, you just need to have one module that contains a list of the packages to flush and the code to do it, and then install that as a PerlCleanupHandler so it will automatically run after every request. - Perrin
Re: PerlRun question
On Mon, 17 Jul 2000, Perrin Harkins wrote: I think it's trying to call a method for an object that doesn't exist anymore. My guess is that I am keeping a certain variable that keeps a list of the object in memory or something similar that I should really be flushing. Because I'm protecting it, the list doesn't get refreshed. Does anyone know the solution to this problem? You may need to modify %INC. Try removing the modules that aren't in your "to keep" list from it at the same point where you clean put their namespaces. PerlRun does a variation of this too. I did what you suggested and I have it clearing out the proper entries in %INC. The problem is that it still doesn't work! Even though they don't exist in the hash, when a cgi needs a module it still tries to call it from memory. Is there maybe another variable that I should be looking at? Andrew
Re: PerlRun question
On Thu, 13 Jul 2000, Perrin Harkins wrote: On Thu, 13 Jul 2000, Andrew Chen wrote: PerlRun already flushes everything, but from my understanding of it, with PerlRunOnce Off mod_perl won't flush the packages that were loaded by the CGIs. What makes you think they need to be flushed? Are they things you wrote in-house that didn't follow the normal module conventions of package names and lexical variables? If not, they shouldn't be a problem. A lot of the code was written without really thinking about potential future moves such as mod_perl. Because of this a lot of variables get used without getting initialized-- that kind of thing. We've already observed some little issues with the code that can be fixed with adding lines to initialize the variables, but that is easier said than done with hundreds (or more) of files. Again, for the reason offered in the first paragraph, we don't want to be cutting and pasting code to flush variables in *every* module we have, because we really have a lot. You can actually do this from one place, iterating through a list of namespaces to flush. Look at the code in Apache::PerlRun::flush_namepsace. It's pretty easy. We would still have to add code to every module, correct? Even though there could be a central variable flushing module, all the other modules would still want to call it. So although we may not be including the flush_namespace code in every module, we will at least need to call them. I look forward to your reply. Thanks for your input on this... Andrew
Re: PerlRun question
On Thu, 13 Jul 2000, Honza Pazdziora wrote: On Thu, Jul 13, 2000 at 01:54:28PM -0700, Andrew Chen wrote: That sounds like a pretty concrete solution. It seemed weird that there wasn't an in-between solution to the issue. The reason probably is, that you really want to clean your code at the end. So all the big guys are either writing everything using Apache::Request, or cleaned up their *.pl's to run fine within Apache::Registry. You need such hackery only for the migration part, not for production code. Yes, ultiimately we are looking to do that. Between the time that it takes for us to set this up and for people to realize how much of a speed boost there is (and thus dedicate resources to it) we need to come up with an interim solution. Thus this these two hacky solutions have been thought up. If you have any other ideas in terms of what another solution for the migration should be, I'd love to hear it. So from your first e-mail, I guess the idea of a module that would be loaded at the end up the startup file that stores away all the information up to that point so that the rest of the stuff loaded after then is flushed-- you've never heard of anything that does something like that? Thanks, Andrew
