You're welcome, Hayden. Glad it helped. Regarding your performance issues, you need to try to understand where the load is coming from. You might just have legitimate traffic, either from users or search bots. Start by looking at your web server logs. We use nginx in front of Tomcat, so I started logging requests to XMLUI, OAI, and REST separately so I could see exactly what was happening in each. You'll get much more insight once you start to understand the nature of your traffic.
For example, Google and Bing respect robots.txt so you if enable the cron job for nightly sitemap generation, then you can submit your sitemap to their web master tools consoles, and then alter your robots.txt so that dynamic search pages like Discovery and Browse are forbidden. This is a big help because those pages are computationally heavy. See this wiki page for how to enable sitemaps: https://wiki.duraspace.org/display/DSDOC5x/Search+Engine+Optimization On that note, from looking at my logs I saw that Baidu doesn't respect robots.txt and just crawls all the dynamic pages. I used nginx to identify Baidu by its user agent and throttle its requests using nginx's limit_req module. I wrote about it on my blog a few months ago: https://mjanja.ch/2017/11/rate-limiting-baiduspider-using-nginx/ Another important tweak along those lines is Tomcat's Crawler Session Manager valve. Basically, Tomcat assigns all users a new session (JSESSIONID) when they connect. Bots are users too, so they get one as well. The problem is that search bots like Google, Bing, and Baidu often crawl from fifty (50) IP addresses concurrently, and each one of those gets a session, each of which takes up precious CPU time, memory, and database resources. If you enable the Crawler Session Manager valve Tomcat will force all matching user agents to use one session. See the Tomcat docs or look at our server.xml for more information: https://github.com/ilri/rmg-ansible-public/blob/master/roles/dspace/templates/tomcat/server-tomcat7.xml.j2 Sometimes I will get an alert that our server's CPU load is very high so I'll go look at the logs and do some grep and awk magic to figure out the IP addresses for the day's top ten users. Often its illuminating. For example, one day in December some client in China downloaded 20,000 PDFs from us. Another time some open access bot started harvesting us and made 400,000 requests in a few hours. I looked at the logs, found the contact information in their user agent string, and contacted their developers. They acknowledged that there bot had gone awry due to a bug and fixed it. Now our repository has been using a pool with a maximum of 300 connections for a month or so and I haven't had DSpace crash due to database issues since then. We still have high load every few days and users complain that the site is "down", but when I look at PostgreSQL activity I see we have 211 active database connections—so there's a problem but it's not the database! This is when I started to look elsewhere in the application stack to alleviate this issue. For example, Tomcat's default maxThreads is 200, so it's likely that this is the bottle neck now. I've recently bumped it and its companion processorCache up to 400 and I'm monitoring things now. So start with looking at the logs! They are very interesting and enlightening. :) Cheers, On Tue, Jan 30, 2018 at 2:11 PM Hayden Young <hay...@knowledgearc.com> wrote: > Hi Alan > > Thanks for the in-depth analysis and possible remedies. I implemented the > jndi connections as you had outlined and tested that it did indeed use the > Tomcat settings and not the DSpace configuration. As soon as I redeployed > our DSpace it certainly felt as if there was an improvement in performance. > A couple of changes were required to the jndi tomcat configuration in > server.xml (probably related to running tomcat8) which were changing > maxWait to maxWaitMillis and maxActive to maxTotal. > > Unfortunately, we were soon back to where we started with > org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for > connection from pool errors when browsing bitstreams. We also noticed the > problem arose quicker than it has in the past; usually it is 1 to 2 weeks > before we see this problem; this time it has happened within 3 days. The > site does not experience much traffic and a check of the Postgres idle > connections shows only 8 which seems well below what would cause the db > error. > > Are you able to outline possible steps to analyzing the problem from > DSpace's end? > > Our other solution is to put PG Bouncer between the db and dspace although > my feeling is that this is only a temporary solution. > > Thanks > > > Hayden Young > https://www.knowledgearc.com > > > On Wednesday, January 17, 2018 at 1:36:16 AM UTC+8, Alan Orth wrote: > >> Thanks, Hardy. I've created a new Jira issue[0], updated my git commit >> message with the DS issue number, and updated my pull request[1] title and >> description to include the fact that this is slightly related to DS-3434. I >> think that's good to go now. >> >> [0] https://jira.duraspace.org/browse/DS-3803 >> [1] https://github.com/DSpace/DSpace/pull/1917 >> >> Cheers, >> >> On Tue, Jan 16, 2018 at 5:41 PM Hardy Pottinger <hardy.p...@gmail.com> >> wrote: >> > Hi, Alan, yes, we require a matching Jira issue for each pull request. >>> When you create a new issue, please mention DS-3434 as being related. >>> Thanks! >>> >>> --Hardy >>> >> On Sat, Jan 13, 2018 at 10:00 PM, Alan Orth <alan...@gmail.com> wrote: >>> >> I've been testing DSpace 6.2 with Tomcat 8.5.24 and noticed that DSpace >>>> does not start if you supply a database pool from JNDI(!). It seems to be a >>>> known issue, as Mark Wood created a Jira ticket for it[0]. Also, DSpace 6.2 >>>> still has the db.jndi setting in dspace.cfg, even though this setting is no >>>> longer user customizable. I've made a pull request against the dspace-6_x >>>> branch to remove it, but I did not create a Jira ticket. Is it required? >>>> >>>> [0] https://jira.duraspace.org/browse/DS-3434 >>>> [1] https://github.com/DSpace/DSpace/pull/1917 >>>> >>>> Regards, >>>> >>> On Thu, Jan 11, 2018 at 9:37 AM Alan Orth <alan...@gmail.com> wrote: >>>> >>> To continue the discussion on a slightly related note: I've just >>>>> finished dealing with the fallout caused by some new bot — the only >>>>> fingerprint of which is its unique-but-normal-looking user agent — hitting >>>>> our XMLUI with 450,000 requests from six different IPs over just a few >>>>> hours. This generated a ridiculous amount of load on the server, including >>>>> 160 PostgreSQL connections and 52,000 Tomcat sessions before I was able to >>>>> mitigate it. Surprisingly, since I had increased out pool size to 300 >>>>> after >>>>> my last message, we never got pool timeout or database connection errors >>>>> in >>>>> dspace.log, but the site was very unresponsive — and this is on a beefy >>>>> server with SSDs, plenty of RAM, large PostgreSQL buffer cache, etc! I >>>>> ended up having to rate limit this user agent in our frontend nginx web >>>>> server using the limit_req_zone module[0]. >>>>> >>>>> So a bit of a mixed success and frustration here. No amount of pool >>>>> tweaking will fix this type of issue, because there's always another >>>>> bigger, stupider bot that comes along eventually and doesn't match the >>>>> "bot" user agent. I will definitely look into implementing separate pools >>>>> as Tom had suggested, though, to limit the damage caused by high load to >>>>> certain DSpace web applications. Keep sharing your experiences! This is >>>>> very valuable and interesting to me. >>>>> >>>>> [0] >>>>> https://github.com/ilri/rmg-ansible-public/commit/368faaa99028c8e0c8a99de3f6c253a228d5f63b >>>>> >>>>> Cheers! >>>>> >>>> On Thu, Jan 4, 2018 at 7:31 AM Alan Orth <alan...@gmail.com> wrote: >>>>> >>>> That's a cool idea to use a separate pool for each web application, >>>>>> Tom! I'd much rather have my OAI fail to establish a database connection >>>>>> than my XMLUI. ;) >>>>>> >>>>>> Since I wrote the original mailing list message two weeks ago I've >>>>>> had DSpace fail to establish a database connection a few thousand times >>>>>> and >>>>>> I've increased my pool's max active from 50 to 75 and then 125 — our site >>>>>> gets about four million hits per month (from looking at nginx logs), so >>>>>> I'm >>>>>> still trying to find the "sweet spot" for the pool settings. Anything's >>>>>> better than setting the pool in dspace.cfg, though. >>>>>> >>>>>> I wish other people would share their pool settings and experiences. >>>>>> >>>>>> On Wed, Jan 3, 2018 at 2:40 PM Hardy Pottinger <hardy.p...@gmail.com> >>>>>> wrote: >>>>>> >>>>> Hi, please do create this wiki page, I'd love to read it. Thanks! >>>>>>> >>>>>>> --Hardy >>>>>>> >>>>>> On Wed, Jan 3, 2018 at 4:10 PM, Tom Desair <tom.d...@atmire.com> >>>>>>> wrote: >>>>>>> >>>>>> I just wanted to add a small note that having 1 single DB pool for >>>>>>>> all Tomcat webapps can (and has) lead to problems. Your current pool >>>>>>>> size >>>>>>>> is 50. This means that if you have (malicious) crawlers hitting your >>>>>>>> OAI >>>>>>>> endpoint, this can deplete the available database connections >>>>>>>> available for >>>>>>>> the web UI (XMLUI or JSPUI). The other way around can also happen. >>>>>>>> >>>>>>>> But using JNDI DB pools also give you more fine-grained control >>>>>>>> over the connection distribution over the different web apps. For >>>>>>>> example, >>>>>>>> a default PostgreSQL installation comes with a max connection limit of >>>>>>>> 100. >>>>>>>> This means you can safely use around 70 connections (from experience). >>>>>>>> You >>>>>>>> can then divided these connections with individual JNDI pools like >>>>>>>> this: >>>>>>>> >>>>>>>> - OAI: 15 connections >>>>>>>> - REST: 15 connections >>>>>>>> - WEB UI: 40 connections >>>>>>>> >>>>>>>> >>>>>>>> Let me know if you've created a JNDI DB pool wiki page. I'll then >>>>>>>> try to add some useful information on JDBC interceptors ( >>>>>>>> https://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html#Configuring_JDBC_interceptors >>>>>>>> ). >>>>>>>> >>>>>>>> >>>>>>>> [image: logo] Tom Desair >>>>>>>> 250-B Suite 3A, Lucius Gordon Drive, West Henrietta, NY 14586 >>>>>>>> <https://maps.google.com/?q=3A,+Lucius+Gordon+Drive,+West+Henrietta,+NY+14586&entry=gmail&source=g> >>>>>>>> Gaston Geenslaan 14, Leuven 3001, Belgium >>>>>>>> <https://maps.google.com/?q=Gaston+Geenslaan+14,+Leuven+3001,+Belgium&entry=gmail&source=g> >>>>>>>> www.atmire.com >>>>>>>> <http://atmire.com/website/?q=services&utm_source=emailfooter&utm_medium=email&utm_campaign=tomdesair> >>>>>>>> >>>>>>> 2018-01-03 22:36 GMT+01:00 Tim Donohue <tdon...@duraspace.org>: >>>>>>>> >>>>>>> Hi Alan & Mark, >>>>>>>>> >>>>>>>>> These notes look like the start to some enhanced documentation >>>>>>>>> around setting up DSpace + Tomcat JNDI (hint, hint). >>>>>>>>> >>>>>>>>> I'm wondering (out loud) if we should take these concepts/ideas >>>>>>>>> and turn them into official documentation in the "Installing DSpace" >>>>>>>>> section (maybe somewhere under "Advanced Installation"?): >>>>>>>>> https://wiki.duraspace.org/display/DSDOC6x/Installing+DSpace >>>>>>>>> >>>>>>>>> Thanks though for sharing the notes and Q&A here. I think this >>>>>>>>> will be very helpful for others who wish to go this route. >>>>>>>>> >>>>>>>>> - Tim >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Jan 3, 2018 at 3:17 PM Mark H. Wood <mwood...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>> Thanks for posting these notes. I'm sure they will be helpful. >>>>>>>>>> You've shown some tools that I didn't know about. >>>>>>>>>> >>>>>>>>>> A pool instantiated by DSpace is probably effectively invisible >>>>>>>>>> to other webapp.s even in the same JVM. The Servlet spec. tries >>>>>>>>>> very hard >>>>>>>>>> to create the illusion that each webapp. is floating in a kind of >>>>>>>>>> womb, >>>>>>>>>> where everything it needs is mysteriously provided for it from >>>>>>>>>> somewhere >>>>>>>>>> beyond its perception. Each has its own classloader, for example, >>>>>>>>>> and so >>>>>>>>>> things that a webapp. creates for itself tend to be known only in >>>>>>>>>> places >>>>>>>>>> that are not accessible by other webapp.s. I could wish that DSpace >>>>>>>>>> made >>>>>>>>>> more thorough use of the Servlet environment rather than behaving as >>>>>>>>>> if it >>>>>>>>>> is standalone code. >>>>>>>>>> >>>>>>>>>> You're quite correct that the command-line tools don't share a >>>>>>>>>> pool with any of the webapp.s, because the launcher runs in a >>>>>>>>>> different >>>>>>>>>> process with its own address space. This is one reason to continue >>>>>>>>>> specifying pool settings in dspace.cfg -- IMO this should be the >>>>>>>>>> *only* use >>>>>>>>>> of those settings. It *is* possible to supply a pool to the command >>>>>>>>>> line >>>>>>>>>> out of JNDI -- I've done it -- but you need to supply a directory >>>>>>>>>> service >>>>>>>>>> to the process. I can say a little about that if anybody is >>>>>>>>>> interested. >>>>>>>>>> You could provide in dspace.cfg settings more appropriate to the >>>>>>>>>> command >>>>>>>>>> line, if your webapp.s are set up with pools (tuned for their needs) >>>>>>>>>> from >>>>>>>>>> JNDI. >>>>>>>>>> >>>>>>>>>> The reason you don't have to tinker with directory services for >>>>>>>>>> webapp.s is that the <Resource> and <ResourceLink> elements are >>>>>>>>>> causing >>>>>>>>>> your Servlet container (Tomcat) to populate an internal directory >>>>>>>>>> service >>>>>>>>>> with objects such as your connection pool. This is specified by >>>>>>>>>> Java EE, >>>>>>>>>> but many Servlet containers implement it even when not required by >>>>>>>>>> the >>>>>>>>>> relevant spec.s. >>>>>>>>>> >>>>>>>>>> You *do* need to supply any DBMS drivers to the container itself, >>>>>>>>>> because the pool and connections are created by the container and so >>>>>>>>>> must >>>>>>>>>> be visible from *its* classloader(s), which (in Tomcat anyway) are >>>>>>>>>> on a >>>>>>>>>> branch of the hierarchy that is parallel to those of the webapp.s. >>>>>>>>>> I also >>>>>>>>>> would use the latest released driver. >>>>>>>>>> >>>>>>>>>> It should be simple to provide a log message when the resolution >>>>>>>>>> of jdbc/dspace *succeeds*, and I think we should. There's already a >>>>>>>>>> Jira >>>>>>>>>> issue about making the other case less scary, and perhaps this >>>>>>>>>> should be >>>>>>>>>> included in that work. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "DSpace Technical Support" group. >>>>>>>>>> >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to dspace-tech...@googlegroups.com. >>>>>>>>>> To post to this group, send email to dspac...@googlegroups.com. >>>>>>>>> >>>>>>>>> >>>>>>>>>> Visit this group at https://groups.google.com/group/dspace-tech. >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Tim Donohue >>>>>>>>> Technical Lead for DSpace & DSpaceDirect >>>>>>>>> DuraSpace.org | DSpace.org | DSpaceDirect.org >>>>>>>>> >>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "DSpace Technical Support" group. >>>>>>>>> >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to dspace-tech...@googlegroups.com. >>>>>>>>> To post to this group, send email to dspac...@googlegroups.com. >>>>>>>>> >>>>>>>> >>>>>>>>> Visit this group at https://groups.google.com/group/dspace-tech. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "DSpace Technical Support" group. >>>>>>>> >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to dspace-tech...@googlegroups.com. >>>>>>>> To post to this group, send email to dspac...@googlegroups.com. >>>>>>>> >>>>>>> >>>>>>>> Visit this group at https://groups.google.com/group/dspace-tech. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "DSpace Technical Support" group. >>>>>>> >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to dspace-tech...@googlegroups.com. >>>>>>> To post to this group, send email to dspac...@googlegroups.com. >>>>>> >>>>>> >>>>>>> Visit this group at https://groups.google.com/group/dspace-tech. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Alan Orth >>>>>> alan...@gmail.com >>>>>> >>>>> >>>>>> https://picturingjordan.com >>>>>> https://englishbulgaria.net >>>>>> https://mjanja.ch >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Alan Orth >>>>> alan...@gmail.com >>>>> >>>> >>>>> https://picturingjordan.com >>>>> https://englishbulgaria.net >>>>> https://mjanja.ch >>>>> >>>> >>>> >>>> -- >>>> >>>> Alan Orth >>>> alan...@gmail.com >>>> >>> >>>> https://picturingjordan.com >>>> https://englishbulgaria.net >>>> https://mjanja.ch >>>> >>> >>> >> >> -- >> >> Alan Orth >> alan...@gmail.com >> > >> https://picturingjordan.com >> https://englishbulgaria.net >> https://mjanja.ch >> > -- > You received this message because you are subscribed to the Google Groups > "DSpace Technical Support" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to dspace-tech+unsubscr...@googlegroups.com. > To post to this group, send email to dspace-tech@googlegroups.com. > Visit this group at https://groups.google.com/group/dspace-tech. > For more options, visit https://groups.google.com/d/optout. > -- Alan Orth alan.o...@gmail.com https://picturingjordan.com https://englishbulgaria.net https://mjanja.ch -- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech+unsubscr...@googlegroups.com. To post to this group, send email to dspace-tech@googlegroups.com. Visit this group at https://groups.google.com/group/dspace-tech. For more options, visit https://groups.google.com/d/optout.
