Hi, Luiz: Yes, I agree with the bot issue and we blocked a few rogue ones (as much as we can identify them). But the problem is different this time: discovery is as fast or even faster because we re-build the Solr index; however, loading a handle has to wait 30 or more secs (no kidding, I see it myself and is true), then loading the pdf and view full metadata record will take 10-20secs each. I think it is related with DB (postgres) because discovery is no problem, so we change the config with maxconnections sets to 95 and maxidle to -1, but it does not help neither. So, my question is, does the problems described above as likely a DSpace (JVM and etc) issue or DB-related (could still related to DSpace code of handling connection pool)?
Thanks -- Hui On Wednesday, June 8, 2016 at 10:01:34 AM UTC-7, Luiz dos Santos wrote: > > Hi Hui, > > I have two comments: > > Were you able to see in the new relic report if there is anything that > the coocon class is wait for such as database or anything else? > Did you already check how many requisition are coming your > repository? We got something like that last year and our problem was that > we were getting too many robots query, it was solved with a filter. > > Best > Luiz > > On Wednesday, June 8, 2016, Hui Zhang <zhan...@gmail.com <javascript:>> > wrote: > >> Thanks Tim. Yeah, I think the next step is to run some tests to find out >> what problem cause the delay beyond Coccon forward. We re-build the >> discovery index and restart the whole server (not just the tomcat), I will >> say it is getting better regarding to speed. We faced similar speed issue >> last year at the same peak weeks, but it gets worse this year and the >> 'root' cause is not clear to us. We probably will run some tests in the >> summer to see whether we can figure it out. >> >> Best, >> -- >> Hui >> >> >> >> >> On Wednesday, June 8, 2016 at 8:11:24 AM UTC-7, Tim Donohue wrote: >>> >>> Hi Hui, >>> >>> This is a new one for me as well. Though, I hope others will speak up if >>> they've seen something similar, as maybe it'd provide us more info around >>> the root cause. >>> >>> As Stuart notes, I'm not sure the CocoonForwardController really is to >>> blame (as it is a simple wrapper), so it might be a matter of digging a bit >>> deeper to see if there are any other clues in your NewRelic monitor. >>> >>> A bottleneck of 33 seconds is a *massive* bottleneck though. Can you >>> "visibly"reproduce the bottleneck in your browser (e.g. does it happen >>> right when you start a new deposit? Is it on a specific step during the >>> deposit process? Is it during the final save/submit?)? Is there any chance >>> you've heavily customized this area of the deposit process (if so, it also >>> may be worth reviewing any custom code you've added to ensure it isn't a >>> possible cause)? >>> >>> As for whether to blame Cocoon, it's really hard to say without more >>> info. Cocoon is a rather outdated platform (which is why we are moving >>> toward a new UI in the near future [1]), and I know others have >>> seen/reported minor "sluggishness" in the past at times. But, nothing to >>> this extent, which makes me wonder if there could be something else going >>> on. >>> >>> If you have a test/development server, it also might be interesting to >>> see if you could reproduce this issue using a tool like JMeter ( >>> http://jmeter.apache.org/). If it's simply grinding to a halt when a >>> large number of users access to the site, you should be able to reproduce >>> that using JMeter (or similar) and then potentially work to narrow down >>> possible causes. For example, see if it is possible to reproduce that same >>> issue on a standard, uncustomized 3.x or 4.x or 5.x DSpace site. >>> >>> I wish I had more hints to share, but it sounds like the next steps >>> would be to try and narrow down the issue, or see if you can reproduce it >>> (via JMeter or similar) so that others can see it in action. Hopefully if >>> anyone else has encountered this, they'll also speak up. >>> >>> Good luck. Let us know if you discover anything else (especially a root >>> cause). >>> >>> Tim >>> >>> [1] >>> https://wiki.duraspace.org/display/DSPACE/DSpace+UI+Prototype+Challenge >>> >>> On 6/7/2016 4:39 PM, Stuart A. Yeates wrote: >>> >>> I believe that you will find that blaming this java class is a >>> measurement artifact. If you look at the code, it's a one-line wrapper >>> around other things: >>> <https://github.com/DSpace/DSpace/blob/master/dspace-xmlui/src/main/java/org/dspace/springmvc/CocoonForwardController.java> >>> https://github.com/DSpace/DSpace/blob/master/dspace-xmlui/src/main/java/org/dspace/springmvc/CocoonForwardController.java >>> >>> >>> As to whether Cocoon is a bottle-neck, that's a deeper question >>> involving how else you'd get certain things done. >>> >>> If I were seeing things grind to a halt while students were submitting >>> their work, I'd look at decoupling the deposit from the rest of repository >>> using the standard SWORD interface. >>> >>> cheers >>> stuart >>> >>> -- >>> ...let us be heard from red core to black sky >>> >>> On Wed, Jun 8, 2016 at 6:45 AM, Hui Zhang <zhan...@gmail.com> wrote: >>> >>>> Hello, >>>> >>>> We have an fairly large these and dissertation IR running on DSpace 3.0 >>>> XMLUI, which faced the problem of sluggish speed recently as students >>>> start >>>> to deposit their works for graduation. Although the usage level is more >>>> than normal, but we still feel it is odd because of the resources >>>> committed >>>> to it (heap memory: 10GB, max allowed DB connections: 80). Our NewRelic >>>> monitor identifies '/CocoonForwardController/forwardRequest' >>>> (org.apache.cocoon.sitemap.SitemapServlet.service()) as the bottleneck >>>> that >>>> takes average 33 sec while others take ms. As Cocoon forwardrequest is >>>> about 98% of the requests, the IR becomes too slow for users. I am >>>> wondering whether other institutes have similar performance issues and >>>> whether Cocoon, or how it is implemented in DSpace, has been discussed >>>> with >>>> speed related issues? We appreciate suggestions and are willing to >>>> collaborate on some solutions as well. >>>> >>>> Best, >>>> -- >>>> Hui Zhang >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "DSpace Technical Support" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to dspace-tech...@googlegroups.com. >>>> To post to this group, send email to dspac...@googlegroups.com. >>>> Visit this group at https://groups.google.com/group/dspace-tech. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "DSpace Technical Support" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to dspace-tech...@googlegroups.com. >>> To post to this group, send email to dspac...@googlegroups.com. >>> Visit this group at https://groups.google.com/group/dspace-tech. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> -- >>> Tim Donohue >>> Technical Lead for DSpace & DSpaceDirect >>> DuraSpace.org | DSpace.org | DSpaceDirect.org >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "DSpace Technical Support" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to dspace-tech+unsubscr...@googlegroups.com. >> To post to this group, send email to dspace-tech@googlegroups.com. >> Visit this group at https://groups.google.com/group/dspace-tech. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech+unsubscr...@googlegroups.com. To post to this group, send email to dspace-tech@googlegroups.com. Visit this group at https://groups.google.com/group/dspace-tech. For more options, visit https://groups.google.com/d/optout.
