Re: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]

[Chris]

On Fri, Feb 12, 2021 at 05:15:29PM +0100, André Warnier (tomcat/perl) wrote:
> My comment was just basically so as to avoid the case where someone else
> would later be searching the archives of this mailing list for information
> about DBI, and never find these (useful for DBI) posts, because DBI is not
> in the subject.

I can't disagree with that. Put up a short thread for the archives with a good
subject and content about what this thread references with some kind of link
to this thread?

Some of the good topics I really found helpful are also long threads
that wander around getting off-topic of the subject. I've saved quite a
few useful posts from those threads. Good archiving is very important.
I have found using regular web search engines to be an exercise in
useless frustration.

I'm reading up on several of the topics here. I plan to ask some more
questions about those after a bit. What is a good way to reference back
to this thread's sections that will get into the archive in a useful way?

Chris


> 
> On 12.02.2021 00:51, Chris wrote:
> > On Thu, Feb 11, 2021 at 09:52:16AM +0100, André Warnier (tomcat/perl) wrote:
> > > Isn't this discussion about connection pools and firewalls etc getting a 
> > > bit
> > > far from the initial subject of the thread ?
> > 
> > Perhaps. But this has become a pretty low volume mailing list.
> > This "thread" has moved me to spend hours looking at changing and/or
> > better understanding the work I have done (pretty old code) and the
> > work I am now starting.
> > 
> > For me, I'm re-reading the manual pages for the DBI modules,
> > etc. I've also added another mailing list to follow about DBI.
> > 
> > And I will now have some threads to add in the near future.
> > Threads I wouldn't have thought of.
> > But this isn't my mailing list, so breaking these topics into new
> > threads is just fine. Not a problem at all. 8-)
> > 
> > Recently, something "clicked on" for me about mod_perl.
> > Which is pretty thrilling for me. ;-}
> > 
> > Chris
> > 
> > 
> > > 
> > > On 09.02.2021 23:03, Mithun Bhattacharya wrote:
> > > > I would consider mine a small setup on an internal network and I have
> > > > used both Sybase and SQL Server. In our case the DBA's preferred us to
> > > > remain connected rather than make too many connections - we need DB
> > > > access in bursts - it could be quiet for more than an hour and then
> > > > suddenly we might need hundreds of connections within few minutes (if we
> > > > didnt cache it). Another thing was we were connecting from forked
> > > > processes so at some point everything gets reaped including the
> > > > connections. Our style of coding has been to connect to the DB wherever
> > > > we actually need to fire one or more SQLs and do connect_cached in the
> > > > actual implementation (it is a separate library since we had to place a
> > > > wrapper to acquire credentials)
> > > > 
> > > > On Tue, Feb 9, 2021 at 2:34 PM James Smith  > > > > wrote:
> > > > 
> > > >  Mithun,
> > > > 
> > > >  I’m not sure on what scale you work – but these are from 
> > > > experience in sites with
> > > >  small to medium load – and we rarely see an appreciable gain in 
> > > > using cached or pooled
> > > >  connections, just the occasional heartache they cause.
> > > >  If you are working on small applications with a minimal number of 
> > > > databases on the DB
> > > >  server then you may see some performance improvement (but tbh not 
> > > > as much as you used
> > > >  to – as the servers have changed) Unfortunately I don’t in both my 
> > > > main and secondary
> > > >  roles, and I know many others who come across these limitations as 
> > > > well.
> > > > 
> > > >  I’m not saying don’t use persistent or cached connections – but 
> > > > leaving it to some
> > > >  hidden layers is not necessarily a good thing to do – it can have 
> > > > unforeseen side
> > > >  effects {and Apache::DBI & PHP pconnect have both shown these up}
> > > > 
> > > >  If you are working with e.g. with MySQL the overhead of the 
> > > > (socket) connection is
> > > >  very small, but having more connections open to cope with 
> > > > persistent connections
> > > >  {memory wise} often needs specifying a much large database server 
> > > > – or not being able
> > > >  to do all the nice tricks to in memory indexes and queries [to 
> > > > increase query
> > > >  performance]. Being able to chose which connections you keep open 
> > > > and which you
> > > >  open/close on a per request basis gives you the benefits of 
> > > > caching without the risks
> > > >  involved [other than the “lock table” issue].
> > > > 
> > > >  __ __
> > > > 
> > > >  __ __
> > > > 
> > > >  *From:*Mithun Bhattacharya  > > > >
> > > >  *Sent:* 09 February 2021 18:34
> > > >  *To:* mod_perl list  > > > >

Re: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]

[tomcat/perl]

My comment was just basically so as to avoid the case where someone else would later be 
searching the archives of this mailing list for information about DBI, and never find 
these (useful for DBI) posts, because DBI is not in the subject.


On 12.02.2021 00:51, Chris wrote:

On Thu, Feb 11, 2021 at 09:52:16AM +0100, André Warnier (tomcat/perl) wrote:

Isn't this discussion about connection pools and firewalls etc getting a bit
far from the initial subject of the thread ?


Perhaps. But this has become a pretty low volume mailing list.
This "thread" has moved me to spend hours looking at changing and/or
better understanding the work I have done (pretty old code) and the
work I am now starting.

For me, I'm re-reading the manual pages for the DBI modules,
etc. I've also added another mailing list to follow about DBI.

And I will now have some threads to add in the near future.
Threads I wouldn't have thought of.
But this isn't my mailing list, so breaking these topics into new
threads is just fine. Not a problem at all. 8-)

Recently, something "clicked on" for me about mod_perl.
Which is pretty thrilling for me. ;-}

Chris




On 09.02.2021 23:03, Mithun Bhattacharya wrote:

I would consider mine a small setup on an internal network and I have
used both Sybase and SQL Server. In our case the DBA's preferred us to
remain connected rather than make too many connections - we need DB
access in bursts - it could be quiet for more than an hour and then
suddenly we might need hundreds of connections within few minutes (if we
didnt cache it). Another thing was we were connecting from forked
processes so at some point everything gets reaped including the
connections. Our style of coding has been to connect to the DB wherever
we actually need to fire one or more SQLs and do connect_cached in the
actual implementation (it is a separate library since we had to place a
wrapper to acquire credentials)

On Tue, Feb 9, 2021 at 2:34 PM James Smith mailto:j...@sanger.ac.uk>> wrote:

 Mithun,

 I’m not sure on what scale you work – but these are from experience in 
sites with
 small to medium load – and we rarely see an appreciable gain in using 
cached or pooled
 connections, just the occasional heartache they cause.
 If you are working on small applications with a minimal number of 
databases on the DB
 server then you may see some performance improvement (but tbh not as much 
as you used
 to – as the servers have changed) Unfortunately I don’t in both my main 
and secondary
 roles, and I know many others who come across these limitations as well.

 I’m not saying don’t use persistent or cached connections – but leaving it 
to some
 hidden layers is not necessarily a good thing to do – it can have 
unforeseen side
 effects {and Apache::DBI & PHP pconnect have both shown these up}

 If you are working with e.g. with MySQL the overhead of the (socket) 
connection is
 very small, but having more connections open to cope with persistent 
connections
 {memory wise} often needs specifying a much large database server – or not 
being able
 to do all the nice tricks to in memory indexes and queries [to increase 
query
 performance]. Being able to chose which connections you keep open and 
which you
 open/close on a per request basis gives you the benefits of caching 
without the risks
 involved [other than the “lock table” issue].

 __ __

 __ __

 *From:*Mithun Bhattacharya mailto:mit...@gmail.com>>
 *Sent:* 09 February 2021 18:34
 *To:* mod_perl list mailto:modperl@perl.apache.org>>
 *Subject:* Re: Moving ExecCGI to mod_perl - performance and custom 
'modules' [EXT]

 __ __

 Connection caching does work for most use cases - we have to accept James 
works in
 scenarios most developers can't fathom :) 

 __ __

 If you are just firing off simple SQL's without any triggers or named 
temporary tables
 involved you should be good. The only times we recall tripping on cached 
connection is
 when two different code snippets tried to create the same temporary table. 
Another
 time the code was expecting the disconnect to complete the connection 
cleanup.

 __ __

 On Tue, Feb 9, 2021 at 11:47 AM Vincent Veyron mailto:vv.li...@wanadoo.fr>> wrote:

 On Sun, 7 Feb 2021 20:21:34 +
 James Smith mailto:j...@sanger.ac.uk>> wrote:

 Hi James,

  > DBI sharing doesn't really gain you much - and can actually lead 
you into a
 whole world of pain. It isn't actually worth turning it on at all.
  >

 Never had a problem with it myself in years of using it, but I wrap my 
queries in
 an eval { } and check $@, so that the scripts are not left hanging; 
also I have a
 postgresql db ;-).

 I ran some tests with ab, I do see an improvement in response speed :

 my $dbh = DBI->connect()
 Concurrency Le