Re: cgi-object not cacheable
On Tue, Nov 13, 2001 at 09:18:04PM -0500, Perrin Harkins wrote:
One run of my script takes about 2 seconds. This includes a lot of
database-queries, calculations and so on. about 0.3 seconds are used
just for one command: $query=new CGI;
That's really awfully slow. Are you positive it's running under mod_perl?
Have you considered using Apache::Request instead of CGI.pm?
its definitely running under mod_perl. But imho the time it takes to
create a new cgi-object should not depend too much wheter its running
under mod_perl or not, cause the CGI-module is loaded before. (In fact
I load in httpd.conf using PerlModule-Directive)
This is not a problem of my persistent variables, cause this works
with many other objects like db-handles (cant use Apache::DBI cause
this keeps to many handles opened, so I need to cache and pool on my
own), filehandles etc.
Whoa, you can't use Apache::DBI but you can cache database handles yourself?
That doesn't make any sense. What are you doing in your caching that's
different from what Apache::DBI does?
This makes very much sense. Apache::DBI does not limit the number of
persistent connections. It just keeps all the connections open per
apache-process. This will sum up to about 20 open
database-connections, each having one postgres-client running 'idle in
transaction' - and my old small serversystem is going weak. So I cant
cache all connections, but only a limited number and so I cache on my
own :) Beside: it is done with a few lines of code, so it wasnt much
work either:
if (exists($ptr-{global}-{dbhandles}-{_some_id_string}))
{
$dbh=$ptr-{global}-{dbhandles}-{_some_id_string};
$dbh or err($ptr,19); # there must have been something wrong internally
if (not $dbh-ping) {$connect=1;$r='reconnect'} # we just reconnect ..
$dbh and $dbh-rollback; # this issue a new begin-transaction and avoid several
problem with 'current_timestamp' that dedlivers the value
# of the time at the beginning of the transaction,
even if this is hours ago. see TROUBLEREPORT1
$r= stored if $r eq '-';
} else {$connect=1;}
if ($connect)
{
$dbh=DBI-connect(connectinformation)
}
and on exit I just disconnect all handles but keeping a specified
amount. I would prefer to handle this in a special pooling-module
like Apache::DBI is, but where one can specify a maximum number of
open connections and a timeout per connection (connection will be
terminated after it was not used a specified amount of time). As soon
as I get IPC::Sharable to work I'll consider writing such a thingy.
best,
peter
--
mag. peter pilsl
phone: +43 676 3574035
fax : +43 676 3546512
email: [EMAIL PROTECTED]
sms : [EMAIL PROTECTED]
pgp-key available
RE: Re: cgi-object not cacheable
Check out the docs at: http://stein.cshl.org/WWW/software/CGI/#mod_perl There's specific notes on how to make CGI.pm play nicely with mod_perl. -- === If you put three drops of poison into a 100 percent pure Java, you get - Windows. If you put a few drops of Java into Windows, you still have Windows. -- Sun Microsystems CEO, Scott McNealy __ Your favorite stores, helpful shopping tools and great gift ideas. Experience the convenience of buying online with Shop@Netscape! http://shopnow.netscape.com/ Get your own FREE, personal Netscape Mail account today at http://webmail.netscape.com/
Re: cgi-object not cacheable
On Wed, Nov 14, 2001 at 10:39:36AM -0500, Perrin Harkins wrote:
its definitely running under mod_perl. But imho the time it takes to
create a new cgi-object should not depend too much wheter its running
under mod_perl or not, cause the CGI-module is loaded before. (In fact
I load in httpd.conf using PerlModule-Directive)
If it was running under CGI, it would be compiling CGI.pm on each request,
which I've seen take .3 seconds. Taking that long just to create the new
CGI instance seems unusual. How did you time it? Are you using
Apache::DProf?
Wouldnt it be compiled at the use-statement ? I timed it using
module-internal loggingfunction which use time::hires.
This makes very much sense. Apache::DBI does not limit the number of
persistent connections. It just keeps all the connections open per
apache-process.
That should mean one connection per process if you're connecting with the
same parameters every time.
in my case it means up to 4 connections per process, cause in fact it
is not one module but 2 (input and output) and each needs to handle 2 different
connections.
if (exists($ptr-{global}-{dbhandles}-{_some_id_string}))
You know that this is only for one process, right? If you limit this cache
to 20 connections, you may get hundreds of connections.
yes, thats why I limit it to 1 or even 0.
I would prefer to handle this in a special pooling-module
like Apache::DBI is, but where one can specify a maximum number of
open connections and a timeout per connection (connection will be
terminated after it was not used a specified amount of time).
You can just set a timeout in your database server. If a connection times
out and then needs to be used, the ping will fail and Apache::DBI will
re-connect.
thats an interesting idea. I experienced crashes on ping to dead
connections under DBD::Pg but this is worth to check.
As soon
as I get IPC::Sharable to work I'll consider writing such a thingy.
You can't share database handles over IPC::Shareable, but you could share a
global number tracking how many total database handles exist. However, I
think you'd be better off using Apache::DBI and limiting the number of
Apache children to the number of connections your database can deal with.
I hope to share databasehandles via IPC. One has to avoid that only
one process writes to a handle at the same time !! (hope I'm right
here) This would offer possibilities to create a pool of handles with
limited max. number and clientsided timeouts. If a process requests a
handle and there is one cached in the pool it will give this handle
back. Otherwise it will create a new handle or - if max. number is
reached - return 0. The calling application can then decide to print
an excuse due to the user 'cause we are so popular we cant server you
:)' or create and destroy a temporary handle to process the request.
This would be something I would actually prefer to Apache::DBI, but I
dont know if its possible, but I'll try. Such a thing would be very
important, especially on slow servers with less ram, where Apache::DBI
opens to many connections in peak-times and leaves the system in a bad
condition ('to many open filehandles')
peter
ps: just if one is interested: today I was very happy to wear a helmet
when I crashed with my bike ;) At least I can write this lines after
my head touched the road. (well : it hurts in the arms when writing
fast ;)
--
mag. peter pilsl
phone: +43 676 3574035
fax : +43 676 3546512
email: [EMAIL PROTECTED]
sms : [EMAIL PROTECTED]
pgp-key available
Re: cgi-object not cacheable
If it was running under CGI, it would be compiling CGI.pm on each
request,
which I've seen take .3 seconds. Taking that long just to create the
new
CGI instance seems unusual. How did you time it? Are you using
Apache::DProf?
Wouldnt it be compiled at the use-statement ?
Yes, but when running under CGI (the protocol, not the module) that use
statement is executed every time.
I timed it using
module-internal loggingfunction which use time::hires.
That's usually pretty accurate, so I guess it really takes that long on your
system. Try Apache::Request! Or even one of the lighter CGI modules like
CGI_Lite.
in my case it means up to 4 connections per process, cause in fact it
is not one module but 2 (input and output) and each needs to handle 2
different
connections.
If you could reduce that, it would certainly help your application's
performance.
I hope to share databasehandles via IPC. One has to avoid that only
one process writes to a handle at the same time !!
IPC::Shareable, and most of the other options, use Storable to serialize
data structures. Storable can't serialize an open socket. You *CAN* share
sockets, but you'd have to write some custom C code to do it. You might
look at the Sybase thing that was posted here recently. (I haven't looked
at it yet, but it sounded interesting.)
if max. number is
reached - return 0. The calling application can then decide to print
an excuse due to the user 'cause we are so popular we cant server you
:)' or create and destroy a temporary handle to process the request.
Even with temporary handles, you have the possibility of all servers being
busy at once and thus using all 4 handles.
This would be something I would actually prefer to Apache::DBI, but I
dont know if its possible, but I'll try. Such a thing would be very
important, especially on slow servers with less ram, where Apache::DBI
opens to many connections in peak-times and leaves the system in a bad
condition ('to many open filehandles')
I still think you'd be better off just limiting the total number of servers
with MaxClients. Put a reverse proxy in front and you'll offload all the
work that doesn't require a database handle.
ps: just if one is interested: today I was very happy to wear a helmet
when I crashed with my bike ;)
I guess my mother was right about that. Keep your helmet on!
Glad you're not dead,
Perrin
RE: cgi-object not cacheable
That's usually pretty accurate, so I guess it really takes
that long on your system. Try Apache::Request! Or even one
of the lighter CGI modules like CGI_Lite.
in my case it means up to 4 connections per process, cause
in fact it
is not one module but 2 (input and output) and each needs
to handle 2
different
connections.
If you could reduce that, it would certainly help your
application's performance.
You should be able to recude that a fair pile. Unless you're connecting to
multiple Boxes, via Multiple DBI-connect statements, you should be able to
piggyback all requests to a given DB server down the one connection.
SELECT fields FROM database.tablename ... I use that a fair bit to avoid
the overhead of keeping an extra connection in Memory.
If it's a function of userids, you should work with your DBA to ensure that
one user account can do what's needed from the pages.
I really can't imagine that you _need_ to connect to 4 different datasources
in most pages.
This would be something I would actually prefer to
Apache::DBI, but I
dont know if its possible, but I'll try. Such a thing
would be very
important, especially on slow servers with less ram, where
Apache::DBI
opens to many connections in peak-times and leaves the
system in a bad
condition ('to many open filehandles')
I still think you'd be better off just limiting the total
number of servers with MaxClients. Put a reverse proxy in
front and you'll offload all the work that doesn't require a
database handle.
I agree with Perrin here. On my systems, the mod_perl processes never have
a peak time where they start openning more connections, since I just
configure Apache to start 110 Processes and never launch more than 110
Processes. The Database always has 110 connections to it.
Just about the only time you *can't* use Apache::DBI is if you're
deliberately re-connecting on page load. Typically the only reason you'd do
this is if you were changing the authentication parameters of the connection
based on the request data. In that case, Apache::DBI would prove no aid.
If you're getting 'too many open filehandles' it sounds like there's some
additional tuning to do on the OS level. FreeBSD needs to be tuned to
properly run mod_perl, I suspect you may need to increase the Maxfiles and
Maxfilesperproc (or whatever OS equiv) on your system.
As Perrin says, just limit the number of httpd-Mod_perl processes, offload
any and all text/images which is not request driven to a proxy system.
--A
attachment: winmail.dat
RE: cgi-object not cacheable
One of the reasons you should probably not have a persistent/global CGI object is that upon a new the CGI module reads in numerous environment variables and setups up its internal structures for that particular query. If $q (in $q=new CGI) was persistent/global you would possibly have the wrong internal data in $q for the current request. -Original Message- From: Peter Pilsl [mailto:[EMAIL PROTECTED]] Sent: Wednesday, 14 November 2001 9:19 AM To: [EMAIL PROTECTED] Subject: cgi-object not cacheable One run of my script takes about 2 seconds. This includes a lot of database-queries, calculations and so on. about 0.3 seconds are used just for one command: $query=new CGI; I tried to cache the retrieved object between several requests by storing to a persistent variable to avoid this long time, but it is not cacheable. (in the meaning of : operations on a cached CGI-object will just produce nothing) This is not a problem of my persistent variables, cause this works with many other objects like db-handles (cant use Apache::DBI cause this keeps to many handles opened, so I need to cache and pool on my own), filehandles etc. any idea ? thnx, peter -- mag. peter pilsl phone: +43 676 3574035 fax : +43 676 3546512 email: [EMAIL PROTECTED] sms : [EMAIL PROTECTED] pgp-key available
Re: cgi-object not cacheable
One run of my script takes about 2 seconds. This includes a lot of database-queries, calculations and so on. about 0.3 seconds are used just for one command: $query=new CGI; That's really awfully slow. Are you positive it's running under mod_perl? Have you considered using Apache::Request instead of CGI.pm? This is not a problem of my persistent variables, cause this works with many other objects like db-handles (cant use Apache::DBI cause this keeps to many handles opened, so I need to cache and pool on my own), filehandles etc. Whoa, you can't use Apache::DBI but you can cache database handles yourself? That doesn't make any sense. What are you doing in your caching that's different from what Apache::DBI does? - Perrin
