Hi Alan Thanks for the in-depth analysis and possible remedies. I implemented the jndi connections as you had outlined and tested that it did indeed use the Tomcat settings and not the DSpace configuration. As soon as I redeployed our DSpace it certainly felt as if there was an improvement in performance. A couple of changes were required to the jndi tomcat configuration in server.xml (probably related to running tomcat8) which were changing maxWait to maxWaitMillis and maxActive to maxTotal.
Unfortunately, we were soon back to where we started with org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool errors when browsing bitstreams. We also noticed the problem arose quicker than it has in the past; usually it is 1 to 2 weeks before we see this problem; this time it has happened within 3 days. The site does not experience much traffic and a check of the Postgres idle connections shows only 8 which seems well below what would cause the db error. Are you able to outline possible steps to analyzing the problem from DSpace's end? Our other solution is to put PG Bouncer between the db and dspace although my feeling is that this is only a temporary solution. Thanks Hayden Young https://www.knowledgearc.com On Wednesday, January 17, 2018 at 1:36:16 AM UTC+8, Alan Orth wrote: > > Thanks, Hardy. I've created a new Jira issue[0], updated my git commit > message with the DS issue number, and updated my pull request[1] title and > description to include the fact that this is slightly related to DS-3434. I > think that's good to go now. > > [0] https://jira.duraspace.org/browse/DS-3803 > [1] https://github.com/DSpace/DSpace/pull/1917 > > Cheers, > > On Tue, Jan 16, 2018 at 5:41 PM Hardy Pottinger <hardy.p...@gmail.com > <javascript:>> wrote: > >> Hi, Alan, yes, we require a matching Jira issue for each pull request. >> When you create a new issue, please mention DS-3434 as being related. >> Thanks! >> >> --Hardy >> >> On Sat, Jan 13, 2018 at 10:00 PM, Alan Orth <alan...@gmail.com >> <javascript:>> wrote: >> >>> I've been testing DSpace 6.2 with Tomcat 8.5.24 and noticed that DSpace >>> does not start if you supply a database pool from JNDI(!). It seems to be a >>> known issue, as Mark Wood created a Jira ticket for it[0]. Also, DSpace 6.2 >>> still has the db.jndi setting in dspace.cfg, even though this setting is no >>> longer user customizable. I've made a pull request against the dspace-6_x >>> branch to remove it, but I did not create a Jira ticket. Is it required? >>> >>> [0] https://jira.duraspace.org/browse/DS-3434 >>> [1] https://github.com/DSpace/DSpace/pull/1917 >>> >>> Regards, >>> >>> On Thu, Jan 11, 2018 at 9:37 AM Alan Orth <alan...@gmail.com >>> <javascript:>> wrote: >>> >>>> To continue the discussion on a slightly related note: I've just >>>> finished dealing with the fallout caused by some new bot — the only >>>> fingerprint of which is its unique-but-normal-looking user agent — hitting >>>> our XMLUI with 450,000 requests from six different IPs over just a few >>>> hours. This generated a ridiculous amount of load on the server, including >>>> 160 PostgreSQL connections and 52,000 Tomcat sessions before I was able to >>>> mitigate it. Surprisingly, since I had increased out pool size to 300 >>>> after >>>> my last message, we never got pool timeout or database connection errors >>>> in >>>> dspace.log, but the site was very unresponsive — and this is on a beefy >>>> server with SSDs, plenty of RAM, large PostgreSQL buffer cache, etc! I >>>> ended up having to rate limit this user agent in our frontend nginx web >>>> server using the limit_req_zone module[0]. >>>> >>>> So a bit of a mixed success and frustration here. No amount of pool >>>> tweaking will fix this type of issue, because there's always another >>>> bigger, stupider bot that comes along eventually and doesn't match the >>>> "bot" user agent. I will definitely look into implementing separate pools >>>> as Tom had suggested, though, to limit the damage caused by high load to >>>> certain DSpace web applications. Keep sharing your experiences! This is >>>> very valuable and interesting to me. >>>> >>>> [0] >>>> https://github.com/ilri/rmg-ansible-public/commit/368faaa99028c8e0c8a99de3f6c253a228d5f63b >>>> >>>> Cheers! >>>> >>>> On Thu, Jan 4, 2018 at 7:31 AM Alan Orth <alan...@gmail.com >>>> <javascript:>> wrote: >>>> >>>>> That's a cool idea to use a separate pool for each web application, >>>>> Tom! I'd much rather have my OAI fail to establish a database connection >>>>> than my XMLUI. ;) >>>>> >>>>> Since I wrote the original mailing list message two weeks ago I've had >>>>> DSpace fail to establish a database connection a few thousand times and >>>>> I've increased my pool's max active from 50 to 75 and then 125 — our site >>>>> gets about four million hits per month (from looking at nginx logs), so >>>>> I'm >>>>> still trying to find the "sweet spot" for the pool settings. Anything's >>>>> better than setting the pool in dspace.cfg, though. >>>>> >>>>> I wish other people would share their pool settings and experiences. >>>>> >>>>> On Wed, Jan 3, 2018 at 2:40 PM Hardy Pottinger <hardy.p...@gmail.com >>>>> <javascript:>> wrote: >>>>> >>>>>> Hi, please do create this wiki page, I'd love to read it. Thanks! >>>>>> >>>>>> --Hardy >>>>>> >>>>>> On Wed, Jan 3, 2018 at 4:10 PM, Tom Desair <tom.d...@atmire.com >>>>>> <javascript:>> wrote: >>>>>> >>>>>>> I just wanted to add a small note that having 1 single DB pool for >>>>>>> all Tomcat webapps can (and has) lead to problems. Your current pool >>>>>>> size >>>>>>> is 50. This means that if you have (malicious) crawlers hitting your >>>>>>> OAI >>>>>>> endpoint, this can deplete the available database connections available >>>>>>> for >>>>>>> the web UI (XMLUI or JSPUI). The other way around can also happen. >>>>>>> >>>>>>> But using JNDI DB pools also give you more fine-grained control over >>>>>>> the connection distribution over the different web apps. For example, a >>>>>>> default PostgreSQL installation comes with a max connection limit of >>>>>>> 100. >>>>>>> This means you can safely use around 70 connections (from experience). >>>>>>> You >>>>>>> can then divided these connections with individual JNDI pools like this: >>>>>>> >>>>>>> - OAI: 15 connections >>>>>>> - REST: 15 connections >>>>>>> - WEB UI: 40 connections >>>>>>> >>>>>>> >>>>>>> Let me know if you've created a JNDI DB pool wiki page. I'll then >>>>>>> try to add some useful information on JDBC interceptors ( >>>>>>> https://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html#Configuring_JDBC_interceptors >>>>>>> ). >>>>>>> >>>>>>> >>>>>>> [image: logo] Tom Desair >>>>>>> 250-B Suite 3A, Lucius Gordon Drive, West Henrietta, NY 14586 >>>>>>> <https://maps.google.com/?q=3A,+Lucius+Gordon+Drive,+West+Henrietta,+NY+14586&entry=gmail&source=g> >>>>>>> Gaston Geenslaan 14, Leuven 3001, Belgium >>>>>>> <https://maps.google.com/?q=Gaston+Geenslaan+14,+Leuven+3001,+Belgium&entry=gmail&source=g> >>>>>>> www.atmire.com >>>>>>> <http://atmire.com/website/?q=services&utm_source=emailfooter&utm_medium=email&utm_campaign=tomdesair> >>>>>>> >>>>>>> 2018-01-03 22:36 GMT+01:00 Tim Donohue <tdon...@duraspace.org >>>>>>> <javascript:>>: >>>>>>> >>>>>>>> Hi Alan & Mark, >>>>>>>> >>>>>>>> These notes look like the start to some enhanced documentation >>>>>>>> around setting up DSpace + Tomcat JNDI (hint, hint). >>>>>>>> >>>>>>>> I'm wondering (out loud) if we should take these concepts/ideas and >>>>>>>> turn them into official documentation in the "Installing DSpace" >>>>>>>> section >>>>>>>> (maybe somewhere under "Advanced Installation"?): >>>>>>>> https://wiki.duraspace.org/display/DSDOC6x/Installing+DSpace >>>>>>>> >>>>>>>> Thanks though for sharing the notes and Q&A here. I think this will >>>>>>>> be very helpful for others who wish to go this route. >>>>>>>> >>>>>>>> - Tim >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jan 3, 2018 at 3:17 PM Mark H. Wood <mwood...@gmail.com >>>>>>>> <javascript:>> wrote: >>>>>>>> >>>>>>>>> Thanks for posting these notes. I'm sure they will be helpful. >>>>>>>>> You've shown some tools that I didn't know about. >>>>>>>>> >>>>>>>>> A pool instantiated by DSpace is probably effectively invisible to >>>>>>>>> other webapp.s even in the same JVM. The Servlet spec. tries very >>>>>>>>> hard to >>>>>>>>> create the illusion that each webapp. is floating in a kind of womb, >>>>>>>>> where >>>>>>>>> everything it needs is mysteriously provided for it from somewhere >>>>>>>>> beyond >>>>>>>>> its perception. Each has its own classloader, for example, and so >>>>>>>>> things >>>>>>>>> that a webapp. creates for itself tend to be known only in places >>>>>>>>> that are >>>>>>>>> not accessible by other webapp.s. I could wish that DSpace made more >>>>>>>>> thorough use of the Servlet environment rather than behaving as if it >>>>>>>>> is >>>>>>>>> standalone code. >>>>>>>>> >>>>>>>>> You're quite correct that the command-line tools don't share a >>>>>>>>> pool with any of the webapp.s, because the launcher runs in a >>>>>>>>> different >>>>>>>>> process with its own address space. This is one reason to continue >>>>>>>>> specifying pool settings in dspace.cfg -- IMO this should be the >>>>>>>>> *only* use >>>>>>>>> of those settings. It *is* possible to supply a pool to the command >>>>>>>>> line >>>>>>>>> out of JNDI -- I've done it -- but you need to supply a directory >>>>>>>>> service >>>>>>>>> to the process. I can say a little about that if anybody is >>>>>>>>> interested. >>>>>>>>> You could provide in dspace.cfg settings more appropriate to the >>>>>>>>> command >>>>>>>>> line, if your webapp.s are set up with pools (tuned for their needs) >>>>>>>>> from >>>>>>>>> JNDI. >>>>>>>>> >>>>>>>>> The reason you don't have to tinker with directory services for >>>>>>>>> webapp.s is that the <Resource> and <ResourceLink> elements are >>>>>>>>> causing >>>>>>>>> your Servlet container (Tomcat) to populate an internal directory >>>>>>>>> service >>>>>>>>> with objects such as your connection pool. This is specified by Java >>>>>>>>> EE, >>>>>>>>> but many Servlet containers implement it even when not required by >>>>>>>>> the >>>>>>>>> relevant spec.s. >>>>>>>>> >>>>>>>>> You *do* need to supply any DBMS drivers to the container itself, >>>>>>>>> because the pool and connections are created by the container and so >>>>>>>>> must >>>>>>>>> be visible from *its* classloader(s), which (in Tomcat anyway) are on >>>>>>>>> a >>>>>>>>> branch of the hierarchy that is parallel to those of the webapp.s. I >>>>>>>>> also >>>>>>>>> would use the latest released driver. >>>>>>>>> >>>>>>>>> It should be simple to provide a log message when the resolution >>>>>>>>> of jdbc/dspace *succeeds*, and I think we should. There's already a >>>>>>>>> Jira >>>>>>>>> issue about making the other case less scary, and perhaps this should >>>>>>>>> be >>>>>>>>> included in that work. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "DSpace Technical Support" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to dspace-tech...@googlegroups.com <javascript:>. >>>>>>>>> To post to this group, send email to dspac...@googlegroups.com >>>>>>>>> <javascript:>. >>>>>>>>> Visit this group at https://groups.google.com/group/dspace-tech. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Tim Donohue >>>>>>>> Technical Lead for DSpace & DSpaceDirect >>>>>>>> DuraSpace.org | DSpace.org | DSpaceDirect.org >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "DSpace Technical Support" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to dspace-tech...@googlegroups.com <javascript:>. >>>>>>>> To post to this group, send email to dspac...@googlegroups.com >>>>>>>> <javascript:>. >>>>>>>> Visit this group at https://groups.google.com/group/dspace-tech. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "DSpace Technical Support" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to dspace-tech...@googlegroups.com <javascript:>. >>>>>>> To post to this group, send email to dspac...@googlegroups.com >>>>>>> <javascript:>. >>>>>>> Visit this group at https://groups.google.com/group/dspace-tech. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "DSpace Technical Support" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to dspace-tech...@googlegroups.com <javascript:>. >>>>>> To post to this group, send email to dspac...@googlegroups.com >>>>>> <javascript:>. >>>>>> Visit this group at https://groups.google.com/group/dspace-tech. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Alan Orth >>>>> alan...@gmail.com <javascript:> >>>>> https://picturingjordan.com >>>>> https://englishbulgaria.net >>>>> https://mjanja.ch >>>>> >>>> >>>> >>>> -- >>>> >>>> Alan Orth >>>> alan...@gmail.com <javascript:> >>>> https://picturingjordan.com >>>> https://englishbulgaria.net >>>> https://mjanja.ch >>>> >>> >>> >>> -- >>> >>> Alan Orth >>> alan...@gmail.com <javascript:> >>> https://picturingjordan.com >>> https://englishbulgaria.net >>> https://mjanja.ch >>> >> >> > > -- > > Alan Orth > alan...@gmail.com <javascript:> > https://picturingjordan.com > https://englishbulgaria.net > https://mjanja.ch > -- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech+unsubscr...@googlegroups.com. To post to this group, send email to dspace-tech@googlegroups.com. Visit this group at https://groups.google.com/group/dspace-tech. For more options, visit https://groups.google.com/d/optout.
