RE: cgi-object not cacheable
One of the reasons you should probably not have a persistent/global CGI object is that upon a "new" the CGI module reads in numerous environment variables and setups up its internal structures for that particular query. If $q (in $q=new CGI) was persistent/global you would possibly have the wrong internal data in $q for the current request. -Original Message- From: Peter Pilsl [mailto:[EMAIL PROTECTED]] Sent: Wednesday, 14 November 2001 9:19 AM To: [EMAIL PROTECTED] Subject: cgi-object not cacheable One run of my script takes about 2 seconds. This includes a lot of database-queries, calculations and so on. about 0.3 seconds are used just for one command: $query=new CGI; I tried to cache the retrieved object between several requests by storing to a persistent variable to avoid this long time, but it is not cacheable. (in the meaning of : operations on a cached CGI-object will just produce nothing) This is not a problem of my persistent variables, cause this works with many other objects like db-handles (cant use Apache::DBI cause this keeps to many handles opened, so I need to cache and pool on my own), filehandles etc. any idea ? thnx, peter -- mag. peter pilsl phone: +43 676 3574035 fax : +43 676 3546512 email: [EMAIL PROTECTED] sms : [EMAIL PROTECTED] pgp-key available
RE: cgi-object not cacheable
> That's usually pretty accurate, so I guess it really takes > that long on your system. Try Apache::Request! Or even one > of the lighter CGI modules like CGI_Lite. > > > in my case it means up to 4 connections per process, cause > in fact it > > is not one module but 2 (input and output) and each needs > to handle 2 > different > > connections. > > If you could reduce that, it would certainly help your > application's performance. You should be able to recude that a fair pile. Unless you're connecting to multiple Boxes, via Multiple DBI->connect statements, you should be able to piggyback all requests to a given DB server down the one connection. SELECT fields FROM database.tablename ... I use that a fair bit to avoid the overhead of keeping an extra connection in Memory. If it's a function of userids, you should work with your DBA to ensure that one user account can do what's needed from the pages. I really can't imagine that you _need_ to connect to 4 different datasources in most pages. > > This would be something I would actually prefer to > Apache::DBI, but I > > dont know if its possible, but I'll try. Such a thing > would be very > > important, especially on slow servers with less ram, where > Apache::DBI > > opens to many connections in peak-times and leaves the > system in a bad > > condition ('to many open filehandles') > > I still think you'd be better off just limiting the total > number of servers with MaxClients. Put a reverse proxy in > front and you'll offload all the work that doesn't require a > database handle. > I agree with Perrin here. On my systems, the mod_perl processes never have a "peak time" where they start openning more connections, since I just configure Apache to start 110 Processes and never launch more than 110 Processes. The Database always has 110 connections to it. Just about the only time you *can't* use Apache::DBI is if you're deliberately re-connecting on page load. Typically the only reason you'd do this is if you were changing the authentication parameters of the connection based on the request data. In that case, Apache::DBI would prove no aid. If you're getting 'too many open filehandles' it sounds like there's some additional tuning to do on the OS level. FreeBSD needs to be tuned to properly run mod_perl, I suspect you may need to increase the Maxfiles and Maxfilesperproc (or whatever OS equiv) on your system. As Perrin says, just limit the number of httpd-Mod_perl processes, offload any and all text/images which is not request driven to a proxy system. --A <>
Re: cgi-object not cacheable
> > If it was running under CGI, it would be compiling CGI.pm on each request, > > which I've seen take .3 seconds. Taking that long just to create the new > > CGI instance seems unusual. How did you time it? Are you using > > Apache::DProf? > > > > Wouldnt it be compiled at the use-statement ? Yes, but when running under CGI (the protocol, not the module) that use statement is executed every time. > I timed it using > module-internal loggingfunction which use time::hires. That's usually pretty accurate, so I guess it really takes that long on your system. Try Apache::Request! Or even one of the lighter CGI modules like CGI_Lite. > in my case it means up to 4 connections per process, cause in fact it > is not one module but 2 (input and output) and each needs to handle 2 different > connections. If you could reduce that, it would certainly help your application's performance. > I hope to share databasehandles via IPC. One has to avoid that only > one process writes to a handle at the same time !! IPC::Shareable, and most of the other options, use Storable to serialize data structures. Storable can't serialize an open socket. You *CAN* share sockets, but you'd have to write some custom C code to do it. You might look at the Sybase thing that was posted here recently. (I haven't looked at it yet, but it sounded interesting.) > if max. number is > reached - return 0. The calling application can then decide to print > an excuse due to the user 'cause we are so popular we cant server you > :)' or create and destroy a temporary handle to process the request. Even with temporary handles, you have the possibility of all servers being busy at once and thus using all 4 handles. > This would be something I would actually prefer to Apache::DBI, but I > dont know if its possible, but I'll try. Such a thing would be very > important, especially on slow servers with less ram, where Apache::DBI > opens to many connections in peak-times and leaves the system in a bad > condition ('to many open filehandles') I still think you'd be better off just limiting the total number of servers with MaxClients. Put a reverse proxy in front and you'll offload all the work that doesn't require a database handle. > ps: just if one is interested: today I was very happy to wear a helmet > when I crashed with my bike ;) I guess my mother was right about that. Keep your helmet on! Glad you're not dead, Perrin
Re: cgi-object not cacheable
On Wed, Nov 14, 2001 at 10:39:36AM -0500, Perrin Harkins wrote: > > its definitely running under mod_perl. But imho the time it takes to > > create a new cgi-object should not depend too much wheter its running > > under mod_perl or not, cause the CGI-module is loaded before. (In fact > > I load in httpd.conf using PerlModule-Directive) > > If it was running under CGI, it would be compiling CGI.pm on each request, > which I've seen take .3 seconds. Taking that long just to create the new > CGI instance seems unusual. How did you time it? Are you using > Apache::DProf? > Wouldnt it be compiled at the use-statement ? I timed it using module-internal loggingfunction which use time::hires. > > This makes very much sense. Apache::DBI does not limit the number of > > persistent connections. It just keeps all the connections open per > > apache-process. > > That should mean one connection per process if you're connecting with the > same parameters every time. > in my case it means up to 4 connections per process, cause in fact it is not one module but 2 (input and output) and each needs to handle 2 different connections. > > if (exists($ptr->{global}->{dbhandles}->{_some_id_string})) > > You know that this is only for one process, right? If you limit this cache > to 20 connections, you may get hundreds of connections. > yes, thats why I limit it to 1 or even 0. > > I would prefer to handle this in a special pooling-module > > like Apache::DBI is, but where one can specify a maximum number of > > open connections and a timeout per connection (connection will be > > terminated after it was not used a specified amount of time). > > You can just set a timeout in your database server. If a connection times > out and then needs to be used, the ping will fail and Apache::DBI will > re-connect. thats an interesting idea. I experienced crashes on ping to dead connections under DBD::Pg but this is worth to check. > > > As soon > > as I get IPC::Sharable to work I'll consider writing such a thingy. > > You can't share database handles over IPC::Shareable, but you could share a > global number tracking how many total database handles exist. However, I > think you'd be better off using Apache::DBI and limiting the number of > Apache children to the number of connections your database can deal with. > I hope to share databasehandles via IPC. One has to avoid that only one process writes to a handle at the same time !! (hope I'm right here) This would offer possibilities to create a pool of handles with limited max. number and clientsided timeouts. If a process requests a handle and there is one cached in the pool it will give this handle back. Otherwise it will create a new handle or - if max. number is reached - return 0. The calling application can then decide to print an excuse due to the user 'cause we are so popular we cant server you :)' or create and destroy a temporary handle to process the request. This would be something I would actually prefer to Apache::DBI, but I dont know if its possible, but I'll try. Such a thing would be very important, especially on slow servers with less ram, where Apache::DBI opens to many connections in peak-times and leaves the system in a bad condition ('to many open filehandles') peter ps: just if one is interested: today I was very happy to wear a helmet when I crashed with my bike ;) At least I can write this lines after my head touched the road. (well : it hurts in the arms when writing fast ;) -- mag. peter pilsl phone: +43 676 3574035 fax : +43 676 3546512 email: [EMAIL PROTECTED] sms : [EMAIL PROTECTED] pgp-key available
RE: Re: cgi-object not cacheable
Check out the docs at: http://stein.cshl.org/WWW/software/CGI/#mod_perl There's specific notes on how to make CGI.pm play nicely with mod_perl. -- === "If you put three drops of poison into a 100 percent pure Java, you get - Windows. If you put a few drops of Java into Windows, you still have Windows." -- Sun Microsystems CEO, Scott McNealy __ Your favorite stores, helpful shopping tools and great gift ideas. Experience the convenience of buying online with Shop@Netscape! http://shopnow.netscape.com/ Get your own FREE, personal Netscape Mail account today at http://webmail.netscape.com/
Re: cgi-object not cacheable
> its definitely running under mod_perl. But imho the time it takes to > create a new cgi-object should not depend too much wheter its running > under mod_perl or not, cause the CGI-module is loaded before. (In fact > I load in httpd.conf using PerlModule-Directive) If it was running under CGI, it would be compiling CGI.pm on each request, which I've seen take .3 seconds. Taking that long just to create the new CGI instance seems unusual. How did you time it? Are you using Apache::DProf? > This makes very much sense. Apache::DBI does not limit the number of > persistent connections. It just keeps all the connections open per > apache-process. That should mean one connection per process if you're connecting with the same parameters every time. > if (exists($ptr->{global}->{dbhandles}->{_some_id_string})) You know that this is only for one process, right? If you limit this cache to 20 connections, you may get hundreds of connections. > I would prefer to handle this in a special pooling-module > like Apache::DBI is, but where one can specify a maximum number of > open connections and a timeout per connection (connection will be > terminated after it was not used a specified amount of time). You can just set a timeout in your database server. If a connection times out and then needs to be used, the ping will fail and Apache::DBI will re-connect. > As soon > as I get IPC::Sharable to work I'll consider writing such a thingy. You can't share database handles over IPC::Shareable, but you could share a global number tracking how many total database handles exist. However, I think you'd be better off using Apache::DBI and limiting the number of Apache children to the number of connections your database can deal with. - Perrin
Re: cgi-object not cacheable
On Tue, Nov 13, 2001 at 09:18:04PM -0500, Perrin Harkins wrote: > > One run of my script takes about 2 seconds. This includes a lot of > > database-queries, calculations and so on. about 0.3 seconds are used > > just for one command: $query=new CGI; > > That's really awfully slow. Are you positive it's running under mod_perl? > Have you considered using Apache::Request instead of CGI.pm? > its definitely running under mod_perl. But imho the time it takes to create a new cgi-object should not depend too much wheter its running under mod_perl or not, cause the CGI-module is loaded before. (In fact I load in httpd.conf using PerlModule-Directive) > > This is not a problem of my persistent variables, cause this works > > with many other objects like db-handles (cant use Apache::DBI cause > > this keeps to many handles opened, so I need to cache and pool on my > > own), filehandles etc. > > Whoa, you can't use Apache::DBI but you can cache database handles yourself? > That doesn't make any sense. What are you doing in your caching that's > different from what Apache::DBI does? This makes very much sense. Apache::DBI does not limit the number of persistent connections. It just keeps all the connections open per apache-process. This will sum up to about 20 open database-connections, each having one postgres-client running 'idle in transaction' - and my old small serversystem is going weak. So I cant cache all connections, but only a limited number and so I cache on my own :) Beside: it is done with a few lines of code, so it wasnt much work either: if (exists($ptr->{global}->{dbhandles}->{_some_id_string})) { $dbh=$ptr->{global}->{dbhandles}->{_some_id_string}; $dbh or err($ptr,19); # there must have been something wrong internally if (not $dbh->ping) {$connect=1;$r='reconnect'} # we just reconnect .. $dbh and $dbh->rollback; # this issue a new begin-transaction and avoid several problem with 'current_timestamp' that dedlivers the value # of the time at the beginning of the transaction, even if this is hours ago. see TROUBLEREPORT1 $r= "stored" if $r eq '-'; } else {$connect=1;} if ($connect) { $dbh=DBI->connect(connectinformation) } and on exit I just disconnect all handles but keeping a specified amount. I would prefer to handle this in a special pooling-module like Apache::DBI is, but where one can specify a maximum number of open connections and a timeout per connection (connection will be terminated after it was not used a specified amount of time). As soon as I get IPC::Sharable to work I'll consider writing such a thingy. best, peter -- mag. peter pilsl phone: +43 676 3574035 fax : +43 676 3546512 email: [EMAIL PROTECTED] sms : [EMAIL PROTECTED] pgp-key available
Re: cgi-object not cacheable
> One run of my script takes about 2 seconds. This includes a lot of > database-queries, calculations and so on. about 0.3 seconds are used > just for one command: $query=new CGI; That's really awfully slow. Are you positive it's running under mod_perl? Have you considered using Apache::Request instead of CGI.pm? > This is not a problem of my persistent variables, cause this works > with many other objects like db-handles (cant use Apache::DBI cause > this keeps to many handles opened, so I need to cache and pool on my > own), filehandles etc. Whoa, you can't use Apache::DBI but you can cache database handles yourself? That doesn't make any sense. What are you doing in your caching that's different from what Apache::DBI does? - Perrin