Re: [ RFC ] New Module Apache::SessionManager
Gerald Richter wrote: Hi Greg, I also hot heard anything back from the poeple I sent a copy to, I can hence only assume that its so good that it's made them speachless ;-) That's more a matter of time, then a matter of speach... I am up against a deadline and hence will be a little short - but this is excellent. Greg, I have taken a look at your module and the code and I think it is really worth publishing it. I have three anonations: 1.) $r-header_out(Location = $r-uri()); Also this code works with most browsers it doesn't conform to the HTTP specs. A location header must include a host part. Shouldn't be to hard add something like $r-header_out(Location = 'http://' . $r - server - server_hostname . $r-uri()); Seems easy - will add it in. 2.) You use global variables for configuration. While this is fast and easy your aren't be able to have different configurations for different directories or virtual hosts. From my point of view this is a must! I want to only run SessionManager for certain directories and are able to use different configurations for different applications. I would highly recomend to use PerlSetVar for configuration. Ok - will require some changes - will add these. 3.) You have a hard coded make_session_id function. This function should be customable i.e. calling a user defined function which returns a new session id and there should be some way to interact with Apache::Session which already have a (configurable) class that generates session ids. Before the session id is send out to the browser, there must be some way to check if it is valid i.e. doesn't already exists, because it can't be (easily) corrected afterwards. I had orgininally though of this, and decided against as this was simpler. My orgininal thoughts were to: a) keep it very small (the whole thing = fast) b) Keep it very simple The logic is that this module just gets the session ID and other handlers / modules authenticate that session ID. If it is wrong then you need to either change the cookie or redirect to a different ID (if no cookies are set). I can see that adding a call back to a user defined subroutine that returns a Session ID would be possible. Adding in a link to Apache::Session would also be possible. As I wrote above I am up against a deadline and will not be able to do anything till next week - I will impliment the above in the order they are writen as they are in simplest - complex ity order. Thanks Gerald for the feedback above - much appreciated (even if there are quite a few good ideas that involve changes !) Thanks again Greg Gerald - Gerald Richterecos electronic communication services gmbh Internetconnect * Webserver/-design/-datenbanken * Consulting Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz E-Mail: [EMAIL PROTECTED] Voice:+49 6133 925151 WWW:http://www.ecos.de Fax: +49 6133 925152 -
Zombies with dbiproxy
Has anyone used dbiproxy extensively? I seem to be getting hundreds of zombie Perl processes (until the proc table is full) after a while. I only replaced my data source string with an equivalent dbi:Proxy:... string and started dbiproxy. Is this a known bug, or is there something I should have done differently? I was trying to reduce the number of active postgres client processes (Apache::DBI created too many) without while still avoiding the postgres connect() overhead. It turned out that the latter was much less severe than the performance problems caused by ~ 100 active postgres client processes, so I stopped using Apache::DBI. Any other suggestions? Regards, Marinos -- Marinos J. Yannikos Preisvergleich Internet Services AG, Linke Wienzeile 4/2/5, A-1060 Wien Tel/Fax: (+431) 5811609-52/-55 == Geizhals-Preisvergleich - jetzt auch für Deutschland: http://geizhals.cc/de/
An idea, for comments
Hi all
Fist of all, sorry for my bad english...
We "think/found" a technic to manage user action through a web
interface. And I like to know your opinion about it.
The goal is to trigger actions through the server without using cgi ( or
mod_perl ) fake pages.
The actions are managed by a Content Handler on a location:
Location /action
An example is better than long ( bad writen text ) so:
/action/Client/Add
Is parsed to obtains an object type ( "Client" ) and an action to
perform ( "Add" )
$object = "Client"
$method = "Add"
and posted data are read in $fdat ( hash table ref )
Then a perl package is found regarding the object type.
$package = MyNameSpace::Client;
An the action is called on that package.
The problem we have was that we wanted the handler to be able to call
any function available on any authorised package, whithout using a
dispatcher
with a long
if() {
}elsif(){
}elsif(){
}else{
}
To sumup, here want we are doing:
1: parsing URI, to obtain a package and a method
2: look if action is allowed on that package, by this user ( with a
session management ).
3: build a string $str = 'MyNameSpace::Client-Add( $session, $fdat
)', $fdat are the data posted throught the handler
4: calling $return = eval $str.
5: find a redirection page for that ( package, action, result )
That all, it's work fine and it's allowing to call package/object methods whithout
having fake pages to maintains.
Comments are very, very welcomes
--
Don't be irreplaceable, if you can't be replaced, you can't be promoted.
Re: An idea, for comments
Hi there, On Fri, 27 Oct 2000, Nouguier wrote: posted data are read in $fdat ( hash table ref ) [snip] 3: build a string $str = 'MyNameSpace::Client-Add( $session, $fdat)', 4: calling $return = eval $str. I hope you are careful with the contents of $fdat! 73, Ged.
Re: [ RFC ] New Module Apache::SessionManager
Greg Cope ([EMAIL PROTECTED]) said something to this effect: $r-header_out(Location = 'http://' . $r - server - server_hostname . $r-uri()); Seems easy - will add it in. It's not that simple, of course -- you need to maintain port numbers and all that. I recommend using Apache::URI -- create a new Apache::URI object, set its attributes from the Apache object, and then call unparse on it. (darren) -- In the fight between you and the world, back the world.
Re: An idea, for comments
You want Apache::Dispatch. It's almost exactly what you are looking for
(and it really rocks, IMHO).
On Fri, 27 Oct 2000, Nouguier wrote:
Hi all
Fist of all, sorry for my bad english...
We "think/found" a technic to manage user action through a web
interface. And I like to know your opinion about it.
The goal is to trigger actions through the server without using cgi ( or
mod_perl ) fake pages.
The actions are managed by a Content Handler on a location:
Location /action
An example is better than long ( bad writen text ) so:
/action/Client/Add
Is parsed to obtains an object type ( "Client" ) and an action to
perform ( "Add" )
$object = "Client"
$method = "Add"
and posted data are read in $fdat ( hash table ref )
Then a perl package is found regarding the object type.
$package = MyNameSpace::Client;
An the action is called on that package.
The problem we have was that we wanted the handler to be able to call
any function available on any authorised package, whithout using a
dispatcher
with a long
if() {
}elsif(){
}elsif(){
}else{
}
To sumup, here want we are doing:
1: parsing URI, to obtain a package and a method
2: look if action is allowed on that package, by this user ( with a
session management ).
3: build a string $str = 'MyNameSpace::Client-Add( $session, $fdat
)', $fdat are the data posted throught the handler
4: calling $return = eval $str.
5: find a redirection page for that ( package, action, result )
That all, it's work fine and it's allowing to call package/object methods whithout
having fake pages to maintains.
Comments are very, very welcomes
--
Don't be irreplaceable, if you can't be replaced, you can't be promoted.
--
Matt/
/||** Director and CTO **
//||** AxKit.com Ltd ** ** XML Application Serving **
// ||** http://axkit.org ** ** XSLT, XPathScript, XSP **
// \\| // ** Personal Web Site: http://sergeant.org/ **
\\//
//\\
// \\
RE: An idea, for comments
-Original Message- From: Matt Sergeant [mailto:[EMAIL PROTECTED]] Sent: Friday, October 27, 2000 7:44 AM To: Nouguier Cc: perl Subject: Re: An idea, for comments You want Apache::Dispatch. It's almost exactly what you are looking for (and it really rocks, IMHO). for anyone who is interested, I'm pretty close to releasing the latest version of Dispatch. here's the Changes so far: - added support for output filtering thanks to Ken Williams for additional Apache::Filter functionality - added FILTERING section to documentation - added DispatchISA functionality to set parent classes for dispatched module - added DispatchAUTOLOAD directive to make AUTOLOAD behavior user defined - added AUTOLOAD section to documentation - more, better documentation - modified Apache::ModuleConfig-get() calls thanks to Doug MacEachern for the enlightenment - fixed bug that allowed for duplicate DispatchExtras directives - fixed dispatch_index bug - fixed one minor "my $x if 0" condition - fixed DispatchStat ISA directive to properly recurse all levels - made mod_perl 1.2401 required due to DIR_MERGE memory leak in 1.24 unfortunately, the filtering capability will not be available until Apache::Filter 1.12 is released (which is incompatable and implemented totally differently from the current Apache::Filter) DispatchISA is my favorite part of the module so far, which allows you to really take advantage of inheritance by specifying any number of (additional) classes all your dispatched classes should inherit from (like a generic Utils package or whatever). It's nothing you can't do by setting @ISA yourself, you can pretty much do everything Dispatch does yourself anyway ;) for anyone who wants a preview, you can find it at http://morpheus.laserlink.net/~gyoung/modules/Apache-Dispatch-0.07.tar.gz comments most welcome... --Geoff
RE: An idea, for comments
On Fri, 27 Oct 2000, Geoffrey Young wrote: - made mod_perl 1.2401 required due to DIR_MERGE memory leak in 1.24 Thats a pretty stern requirement at least until 1.25 is released. Perhaps you could do what I intend to do with AxKit - do a regex which renames DIR_MERGE to DISABLED_DIR_MERGE when running under 1.24 (needs to be done physically in the file when you run Makefile.PL otherwise it will crash). -- Matt/ /||** Director and CTO ** //||** AxKit.com Ltd ** ** XML Application Serving ** // ||** http://axkit.org ** ** XSLT, XPathScript, XSP ** // \\| // ** Personal Web Site: http://sergeant.org/ ** \\// //\\ // \\
RE: An idea, for comments
-Original Message- From: Matt Sergeant [mailto:[EMAIL PROTECTED]] Sent: Friday, October 27, 2000 8:04 AM To: Geoffrey Young Cc: Nouguier; '[EMAIL PROTECTED]' Subject: RE: An idea, for comments On Fri, 27 Oct 2000, Geoffrey Young wrote: - made mod_perl 1.2401 required due to DIR_MERGE memory leak in 1.24 Thats a pretty stern requirement at least until 1.25 is released. Perhaps you could do what I intend to do with AxKit - do a regex which renames DIR_MERGE to DISABLED_DIR_MERGE when running under 1.24 (needs to be done physically in the file when you run Makefile.PL otherwise it will crash). that sounds interesting - I'd like to see what you have so far... I'm not sure I'll go that route, though - maybe if it relied on cvs... but since 1.24-01 is an official release, I don't see any harm in requiring it. Doug had said that it's pretty close to being 1.25 and there's lots more bug fixes that are important in that release as well (though DIR_MERGE is the only one that affects Dispatch I think)... besides, I'm waiting to hear from Ken on an ETA on the new Apache::Filter - if it's too far away I'll comment out the filtering stuff and release 0.07, otherwise I'll probably wait on that - I'm starting to get on a Apache::Compress kick :) --Geoff -- Matt/ /||** Director and CTO ** //||** AxKit.com Ltd ** ** XML Application Serving ** // ||** http://axkit.org ** ** XSLT, XPathScript, XSP ** // \\| // ** Personal Web Site: http://sergeant.org/ ** \\// //\\ // \\
Re: Connection Pooling / TP Monitor
"Tim" == Tim Bunce [EMAIL PROTECTED] writes: Tim You could have a set of apache servers that are 'pure' DBI proxy Tim servers. That is, they POST requests containing SQL (for Tim prepare_cached) plus bind parameter values and return responses Tim containing the results. Tim Basically I'm proposing that apache be used as an alternative Tim framework for DBI::ProxyServer. Almost all the marshaling code Tim and higher level logic is already in DBI::ProxyServer and Tim DBD::Proxy. Shouldn't be too hard to do and you'd gain in all Tim sorts of ways. You could also use SOAP or SOAP::Lite as the interface. Most of that code seems ready for this kind of application already. -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 [EMAIL PROTECTED] URL:http://www.stonehenge.com/merlyn/ Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
Re: Connection Pooling / TP Monitor
On 27 Oct 2000, (Randal L. Schwartz) wrote: "Tim" == Tim Bunce [EMAIL PROTECTED] writes: Tim You could have a set of apache servers that are 'pure' DBI proxy Tim servers. That is, they POST requests containing SQL (for Tim prepare_cached) plus bind parameter values and return responses Tim containing the results. Tim Basically I'm proposing that apache be used as an alternative Tim framework for DBI::ProxyServer. Almost all the marshaling code Tim and higher level logic is already in DBI::ProxyServer and Tim DBD::Proxy. Shouldn't be too hard to do and you'd gain in all Tim sorts of ways. You could also use SOAP or SOAP::Lite as the interface. Most of that code seems ready for this kind of application already. There are some issues still with this architecture, the primary one is that SOAP is too heavy weight for anything that seriously needs connection pooling for speed issues, especially in Perl (due to the XML parsing speed issues). -- Matt/ /||** Director and CTO ** //||** AxKit.com Ltd ** ** XML Application Serving ** // ||** http://axkit.org ** ** XSLT, XPathScript, XSP ** // \\| // ** Personal Web Site: http://sergeant.org/ ** \\// //\\ // \\
UK based mod_perl/Apache/MySQL type available
Chaps, I'm looking for a couple of weeks good hard heavy hacking in the above skills areas. Short term CTO work (which anyone using the above probably doesn't need) is good too. Work done, happy customer references and rates supplied on demand. Where I add value the most IMHO, is in the application of Stas' stuff - layering, load balancing, performance tuning and scaling. Make that site fly :-) End of advert, back to your regularly scheduled Monday night football game... Dave -- Dave Hodgkinson, http://www.hodgkinson.org Editor-in-chief, The Highway Star http://www.deep-purple.com Apache, mod_perl, MySQL, Sybase hired gun for, well, hire -
RE: An idea, for comments
This is very similar to SOAP (Simple Object Access Protocol). There is a perl
module that implements SOAP. It's also like the many perl RPC modules.
On 27-Oct-2000 Nouguier wrote:
Hi all
Fist of all, sorry for my bad english...
We "think/found" a technic to manage user action through a web
interface. And I like to know your opinion about it.
The goal is to trigger actions through the server without using cgi ( or
mod_perl ) fake pages.
The actions are managed by a Content Handler on a location:
Location /action
An example is better than long ( bad writen text ) so:
/action/Client/Add
Is parsed to obtains an object type ( "Client" ) and an action to
perform ( "Add" )
$object = "Client"
$method = "Add"
and posted data are read in $fdat ( hash table ref )
Then a perl package is found regarding the object type.
$package = MyNameSpace::Client;
An the action is called on that package.
The problem we have was that we wanted the handler to be able to call
any function available on any authorised package, whithout using a
dispatcher
with a long
if() {
}elsif(){
}elsif(){
}else{
}
To sumup, here want we are doing:
1: parsing URI, to obtain a package and a method
2: look if action is allowed on that package, by this user ( with a
session management ).
3: build a string $str = 'MyNameSpace::Client-Add( $session, $fdat
)', $fdat are the data posted throught the handler
4: calling $return = eval $str.
5: find a redirection page for that ( package, action, result )
That all, it's work fine and it's allowing to call package/object methods
whithout having fake pages to maintains.
Comments are very, very welcomes
--
Don't be irreplaceable, if you can't be replaced, you can't be promoted.
--
Jason Bodnar
[EMAIL PROTECTED]
Re: ApacheCon report
On Fri, 27 Oct 2000, Tim Sweetman wrote: In no particular order, and splitting hairs some of the time... Sounded like mod_backhand was best used NOT in the same Apache as a phat application server (eg. mod_perl), because you don't want memory-heavy processes sitting waiting for responses. You'd be better off with a separate switching machine - or serve your static content from machine(s) that know to backhand dynamic requests to a phat machine. I think that's what Theo reckoned... Yes, but the backend mod_perl servers are running backhand. So you have: B B B B \ | | / \ \/ / \|/ F Where all the servers are running mod_backhand, but only F is publically accessible. There may also be 1 F. Its in his slides, and is prettier than the above :-) "make simple things easy, and hard things possible" - What concerns me about systems like AxKit Cocoon is that they may make simple things complex, and some hard things possible. But this is a naive comment not based on trying to build rilly big systems with them. Perl, maybe, doesn't make simple things anything like as easy as PHP. (Again, a naive comment that may be incorrect) No, its correct, I think. I'd like to maybe next time do the second half of the AxKit stuff as a Demo, but that takes some demo-able material thats actually going to make you say "Ooh, that *is* easier than what I'm using right now". So I'll work on it :-) Douglas Adams, who spoke at ApacheCon, previously made an interesting BBC documentary on hypermedia its possibilities, in about 1992. Ted Nelson, I think it was, realised that the ability to _include stuff from other sources in your documents_ was important, and called it a "transclusion" (though that concept, IIRC, may have included the propagation of nanopayments to the source - not sure). And at Apachecon, the XML guys say: "This Document() function's really cool! You can build a portal very easily..." And after falling asleep (reflex on hearing /portal/, marketing allergy) I thought, it's syndication/transclusion again. Evidently, a big idea. But a big idea buried in a heap of other big ideas. Its all been done before. I spoke a bit to Rael Dornfest about P2P (Rael is an O'Reilly guy behind the P2P summit). Basically its all the same stuff we've already been doing, but its just a buzzword. But it often takes buzzwords to make the world sit up and take notice and focus on the right thing to be doing, even though they may not know it themselves! -- Matt/ /||** Director and CTO ** //||** AxKit.com Ltd ** ** XML Application Serving ** // ||** http://axkit.org ** ** XSLT, XPathScript, XSP ** // \\| // ** Personal Web Site: http://sergeant.org/ ** \\// //\\ // \\
RE: ApacheCon report
Do you happen to have the URL for Theo's presentation? I don't see it on the apachecon site. Many thanks, Dave Waldo -Original Message- From: Matt Sergeant [mailto:[EMAIL PROTECTED]] Sent: Friday, October 27, 2000 12:37 PM To: Tim Sweetman Cc: [EMAIL PROTECTED] Subject: Re: ApacheCon report Yes, but the backend mod_perl servers are running backhand. So you have: B B B B \ | | / \ \/ / \|/ F Where all the servers are running mod_backhand, but only F is publically accessible. There may also be 1 F. Its in his slides, and is prettier than the above :-)
RE: ApacheCon report
-Original Message- From: David Waldo [mailto:[EMAIL PROTECTED]] Sent: Friday, October 27, 2000 12:53 PM To: [EMAIL PROTECTED] Subject: RE: ApacheCon report Do you happen to have the URL for Theo's presentation? I don't see it on the apachecon site. http://www.backhand.org/ Many thanks, Dave Waldo -Original Message- From: Matt Sergeant [mailto:[EMAIL PROTECTED]] Sent: Friday, October 27, 2000 12:37 PM To: Tim Sweetman Cc: [EMAIL PROTECTED] Subject: Re: ApacheCon report Yes, but the backend mod_perl servers are running backhand. So you have: B B B B \ | | / \ \/ / \|/ F Where all the servers are running mod_backhand, but only F is publically accessible. There may also be 1 F. Its in his slides, and is prettier than the above :-)
RE: ApacheCon report
-Original Message- From: Matt Sergeant [mailto:[EMAIL PROTECTED]] Sent: Friday, October 27, 2000 12:37 PM To: Tim Sweetman Cc: [EMAIL PROTECTED] Subject: Re: ApacheCon report On Fri, 27 Oct 2000, Tim Sweetman wrote: In no particular order, and splitting hairs some of the time... Sounded like mod_backhand was best used NOT in the same Apache as a phat application server (eg. mod_perl), because you don't want memory-heavy processes sitting waiting for responses. You'd be better off with a separate switching machine - or serve your static content from machine(s) that know to backhand dynamic requests to a phat machine. I think hat's what Theo reckoned... Yes, but the backend mod_perl servers are running backhand. So you have: B B B B \ | | / \ \/ / \|/ F I was really impressed with backhand at Theo's presentation at ApacheCon US in March. From what I rememeber though, it had serious limitations in the SSL space. Did Theo touch on that? The converstation I had with him about it back then was that it was going to be addressed in a future release... also IIRC, backhand was only terribly useful behind something like BigIP (which is what we use). Is there another implementation sheme now? perhaps my memory is fading... --Geoff
RE: ApacheCon report
On Fri, 27 Oct 2000, Geoffrey Young wrote: I was really impressed with backhand at Theo's presentation at ApacheCon US in March. From what I rememeber though, it had serious limitations in the SSL space. Did Theo touch on that? The converstation I had with him about it back then was that it was going to be addressed in a future release... Yes he did touch on that, but I wasn't really understanding what he was saying. I think it was that the frontend server is SSL enabled only, and so the backend servers don't get the SSL cert. But he said you could use a module to put the cert in a header, or something like that, and it would work fine so long as you programmed it all right :-) But then I know very little about SSL, and maybe got the wrong idea. also IIRC, backhand was only terribly useful behind something like BigIP (which is what we use). Is there another implementation sheme now? No that's right. Its the difference between high availability (which BigIP *can* do) and load balancing (which backhand does). -- Matt/ /||** Director and CTO ** //||** AxKit.com Ltd ** ** XML Application Serving ** // ||** http://axkit.org ** ** XSLT, XPathScript, XSP ** // \\| // ** Personal Web Site: http://sergeant.org/ ** \\// //\\ // \\
Re: how to really bang on a script?
Hi there, On Fri, 27 Oct 2000, Christopher L. Everett wrote: I've written some mod_perl scripts that need testing over a million hits or so before I deploy it. ab (distributed with Apache, 'man ab' for help) can give you a million hits with one command. I don't know if you're going to get a real-life test other than by going live. Scripts tend to test what you think will go wrong. Invariably something else goes wrong in real life. Putting a million records in a logfile will test your disc capacity and little else. I need to prove to myself and my marketing guy that my script has certain statistical properties, not the least of which is the question of whether my activity logs match what actually happened. You've been spending too much time with your marketing guy. "Certain statistical properties" is gobbledygook. What properties? Activity logs don't match statistically. Either they match or they don't. If they don't then either the logging is turned off or it isn't working. Also, there's concurrency issues to make sure I've got right. Get yourself a bunch of users. Probably the easiest way to really bang on your script would be to advertise a new sex site (sorry:). Anyway I'd really hate you to do all that work only to find that you hadn't tested the test script well enough... 73, Ged.
Re: how to really bang on a script?
"G.W. Haywood" wrote: Hi there, On Fri, 27 Oct 2000, Christopher L. Everett wrote: snipped helpful advice I need to prove to myself and my marketing guy that my script has certain statistical properties, not the least of which is the question of whether my activity logs match what actually happened. You've been spending too much time with your marketing guy. "Certain statistical properties" is gobbledygook. What properties? Activity logs don't match statistically. Either they match or they don't. If they don't then either the logging is turned off or it isn't working. OK, I confess: I've written (probably yet another) mod_perl banner exchange. I need to know that when we serve 100K banners to 40K different IP addresses a day, and we are selling 30K banners/day, the 500 sites that hosted our banner ads are getting the 50K banners that we promised them, and the 30K banners that we sold, we really did serve. Also, I want to know that the banners my logs say the script sent are really the ones people saw on their browsers. That's what I meant by "certain statistical properties". So, I apologize for not describing my problem clearly in the first place. And again, my questions are: How would I go about proving to myself that my script does what I designed it to do? Has anyone else dealt with a similar problem, and how did they go about doing it? If I solve it for myself, would anyone else find the solution useful, and how would I make it more useful to them? Usually, I would test by running through the script a few times with some variations, but we are so freaked out by our experience with the 2 other banner exchange scripts we tried, we find a lot of value in being certain. Thanks again for your kind help. --Christopher Everett
Re: Connection Pooling / TP Monitor
The only way I really see this working is in a threading environment. First of all, for some databases database connections don't survive forking (Oracle is the notable example here). Also, even if we could get forking to work, we would still get the scaling problem we are trying to avoid. Instead of Oracle keeping a huge list of persistent connections, the Connection Pool would keep a huge list of persistent connections. In both cases each connection would map to a Unix process and all these processes would chew up OS resources big time! - Original Message - From: "Matt Sergeant" [EMAIL PROTECTED] To: "Tim Bunce" [EMAIL PROTECTED] Cc: [EMAIL PROTECTED]; "Jeff Horn" [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Friday, October 27, 2000 7:02 AM Subject: Re: Connection Pooling / TP Monitor On Fri, 27 Oct 2000, Tim Bunce wrote: Sounds like just a CORBA/RPC type thing. Wouldn't you be better off using CORBA::ORBit? Maybe. I dunno. I don't actually need this stuff, I just want there to be a solution out there for those that do. I'm waving my hands around and pointing in various directions hoping someone will _do_ something! Hehe... OK, lets think about exactly what is needed here then. I figure Doug's Apache::DBIPool module (for mod_perl 2.0) is exactly the right architecture: 2 pools of connections (Busy and Waiting) New connections always taken from the head of Waiting Finished connections always replaced on the head of Waiting Threaded architecture (DBI::Oracle handles don't survive a fork) One thread for management One thread per connection once a handle has been supplied Some sort of timeout mechanism for connections if the pool is fully allocated Anything I've missed? If we don't go the threaded route, we can't easily expand and contract the connection pool I don't think - but I'd love to be proved wrong. Also an entire Apache server for the connection pool would be too much - the pre-forking server from the cookbook would be better. And it should even work on Win32 now... -- Matt/ /||** Director and CTO ** //||** AxKit.com Ltd ** ** XML Application Serving ** // ||** http://axkit.org ** ** XSLT, XPathScript, XSP ** // \\| // ** Personal Web Site: http://sergeant.org/ ** \\// //\\ // \\
Re: Connection Pooling / TP Monitor
On Fri, 27 Oct 2000, Jeff Horn wrote: The only way I really see this working is in a threading environment. First of all, for some databases database connections don't survive forking (Oracle is the notable example here). Also, even if we could get forking to work, we would still get the scaling problem we are trying to avoid. Instead of Oracle keeping a huge list of persistent connections, the Connection Pool would keep a huge list of persistent connections. In both cases each connection would map to a Unix process and all these processes would chew up OS resources big time! I don't think thats a 100% fair analysis. First off, forking doesn't use any more RAM than threading until you start copying data. And I don't see that much of a scaling issue - we'd have a pool of connections available, each as a child. Since you could control the number of connections produced you wouldn't have to worry about the scalability issues too much - the problems with things like mod_perl are that there is one connection per child, regardless of what that child is doing. A pool would allow you to make that a lot cleaner, and less resource intensive. At least I think so... Unfortunately I don't have a need for this right now, so I'm not willing to put the hacking tuits into it. Sorry :-( -- Matt/ /||** Director and CTO ** //||** AxKit.com Ltd ** ** XML Application Serving ** // ||** http://axkit.org ** ** XSLT, XPathScript, XSP ** // \\| // ** Personal Web Site: http://sergeant.org/ ** \\// //\\ // \\
Re: Connection Pooling / TP Monitor
On Fri, 27 Oct 2000, Tim Bunce wrote: On Thu, Oct 26, 2000 at 08:47:20PM +0100, Matt Sergeant wrote: On Tue, 24 Oct 2000, Jeff Horn wrote: However, I am also aware of a _major_ ISP that implements their email system using a _major_ RDBMS that has had problems that are best solved via connection pooling. Essentially, the time it takes them to search through all the cached connections is nearly as long as the time it is taking to read/write to the database. Although, I'm not implementing email as this ISP is, I think that scalability in my case may definitely run into similar roadblocks. I am interested in hearing from anyone that has tried to implement true connection pooling either within Apache or as an external process. I'm particularly interested in hearing about implementations that could be made to work or are done using Perl and DBI/DBD. I am mostly interested in things that are Open Source or licensed like Perl itself. Having just returned from ApacheCon, I can honestly recommend looking at mod_backhand to simply have a few servers that run the DBI pool, and have database intensive requests go to those servers. It is a *very* cool solution to just these sorts of scalability problems. To redirect incoming url's that require database work to mod_perl 'heavy' servers? Just like a smarter and more dynamic mod_rewrite? Yes? Yes basically, except its not a redirect. mod_backhand can use keep-alives to ensure that it never has to recreate a new connection to the heavy backend servers, unlike mod_rewrite or mod_proxy. And it can do it in a smart way so that remote connections don't use keepalives (because they are evil for mod_perl servers - see the mod_perl guide), but backhand connections do. Very very cool technology. Or, here's an odd thought that just crossed my mind... You could have a set of apache servers that are 'pure' DBI proxy servers. That is, they POST requests containing SQL (for prepare_cached) plus bind parameter values and return responses containing the results. Basically I'm proposing that apache be used as an alternative framework for DBI::ProxyServer. Almost all the marshaling code and higher level logic is already in DBI::ProxyServer and DBD::Proxy. Shouldn't be too hard to do and you'd gain in all sorts of ways. Anyone fancy having a go? Let me know so we can discuss it in more detail. Sounds like just a CORBA/RPC type thing. Wouldn't you be better off using CORBA::ORBit? -- Matt/ /||** Director and CTO ** //||** AxKit.com Ltd ** ** XML Application Serving ** // ||** http://axkit.org ** ** XSLT, XPathScript, XSP ** // \\| // ** Personal Web Site: http://sergeant.org/ ** \\// //\\ // \\
Re: Connection Pooling / TP Monitor
On Fri, Oct 27, 2000 at 12:26:44PM +0100, Matt Sergeant wrote: Or, here's an odd thought that just crossed my mind... You could have a set of apache servers that are 'pure' DBI proxy servers. That is, they POST requests containing SQL (for prepare_cached) plus bind parameter values and return responses containing the results. Basically I'm proposing that apache be used as an alternative framework for DBI::ProxyServer. Almost all the marshaling code and higher level logic is already in DBI::ProxyServer and DBD::Proxy. Shouldn't be too hard to do and you'd gain in all sorts of ways. Anyone fancy having a go? Let me know so we can discuss it in more detail. Sounds like just a CORBA/RPC type thing. Wouldn't you be better off using CORBA::ORBit? Maybe. I dunno. I don't actually need this stuff, I just want there to be a solution out there for those that do. I'm waving my hands around and pointing in various directions hoping someone will _do_ something! :-) Tim.
Re: Connection Pooling / TP Monitor
At 03:41 PM 10/27/00 +0100, Matt Sergeant wrote: On 27 Oct 2000, (Randal L. Schwartz) wrote: "Tim" == Tim Bunce [EMAIL PROTECTED] writes: Tim You could have a set of apache servers that are 'pure' DBI proxy Tim servers. That is, they POST requests containing SQL (for Tim prepare_cached) plus bind parameter values and return responses Tim containing the results. Tim Basically I'm proposing that apache be used as an alternative Tim framework for DBI::ProxyServer. Almost all the marshaling code Tim and higher level logic is already in DBI::ProxyServer and Tim DBD::Proxy. Shouldn't be too hard to do and you'd gain in all Tim sorts of ways. You could also use SOAP or SOAP::Lite as the interface. Most of that code seems ready for this kind of application already. There are some issues still with this architecture, the primary one is that SOAP is too heavy weight for anything that seriously needs connection pooling for speed issues, especially in Perl (due to the XML parsing speed issues). What we did for our SOAP objects is that we don't use XML Parsing exactly. The reality is that SOAP parsing with a generic object library is heavy weight. But if you are supporting only 7-8 method calls, it's really not bad to write regex that can do all the appropriate parsing with very little code. Our SOAP drivers have well-defined interfaces and are extremely fast using IO::Socket, Regex SOAP utility methods (to construct and strip things like envelope headers). You might argue that there are more things to go wrong this way and you would be right. However, method calls are usually quite well defined and always have the same basic parameter definitions. So as long as you stick to the well-defined interfaces, it's not bad in practice.
Re: Connection Pooling / TP Monitor
On Fri, 27 Oct 2000, Tim Bunce wrote: Sounds like just a CORBA/RPC type thing. Wouldn't you be better off using CORBA::ORBit? Maybe. I dunno. I don't actually need this stuff, I just want there to be a solution out there for those that do. I'm waving my hands around and pointing in various directions hoping someone will _do_ something! Hehe... OK, lets think about exactly what is needed here then. I figure Doug's Apache::DBIPool module (for mod_perl 2.0) is exactly the right architecture: 2 pools of connections (Busy and Waiting) New connections always taken from the head of Waiting Finished connections always replaced on the head of Waiting Threaded architecture (DBI::Oracle handles don't survive a fork) One thread for management One thread per connection once a handle has been supplied Some sort of timeout mechanism for connections if the pool is fully allocated Anything I've missed? If we don't go the threaded route, we can't easily expand and contract the connection pool I don't think - but I'd love to be proved wrong. Also an entire Apache server for the connection pool would be too much - the pre-forking server from the cookbook would be better. And it should even work on Win32 now... -- Matt/ /||** Director and CTO ** //||** AxKit.com Ltd ** ** XML Application Serving ** // ||** http://axkit.org ** ** XSLT, XPathScript, XSP ** // \\| // ** Personal Web Site: http://sergeant.org/ ** \\// //\\ // \\
Re: Connection Pooling / TP Monitor
I would second that. We've done this using SOAP. We have a DataSource::SOAP driver that acts as a lightweight interface to a Jakarta TomCat server for the DB stuff. We get the benefits of Perl on the front-end and Java DB Connection pooling logic/proxying on the middle tier. Of course I guess you could do the SOAP Server in Perl too, but Java was a bit easier because we also get built in shared memory caching for frequently issued queries with the way our particular interfaces work. Anyway, with SOAP it doesn't matter what language you use for what -- in theory. Later, Gunther At 07:38 AM 10/27/00 -0700, Randal L. Schwartz wrote: "Tim" == Tim Bunce [EMAIL PROTECTED] writes: Tim You could have a set of apache servers that are 'pure' DBI proxy Tim servers. That is, they POST requests containing SQL (for Tim prepare_cached) plus bind parameter values and return responses Tim containing the results. Tim Basically I'm proposing that apache be used as an alternative Tim framework for DBI::ProxyServer. Almost all the marshaling code Tim and higher level logic is already in DBI::ProxyServer and Tim DBD::Proxy. Shouldn't be too hard to do and you'd gain in all Tim sorts of ways. You could also use SOAP or SOAP::Lite as the interface. Most of that code seems ready for this kind of application already. -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 [EMAIL PROTECTED] URL:http://www.stonehenge.com/merlyn/ Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training! __ Gunther Birznieks ([EMAIL PROTECTED]) eXtropia - The Web Technology Company http://www.extropia.com/
