Re: [Dspace-tech] FW: Java heap space error when trying to update a Collection
Elliot Metsger wrote: > We've noticed issues with heap size as well, specifically when using the > wildcard policy tool, the media filter manager, and some in-house tools that > iterate over dspace objects using a single Context. > > When dspace objects like Item or Bitstream are instantiated, they place a > reference to itself in a hash map attached to the context. So over time > that map will get large. We modified the tools to clear the context "cache" > when they were done operating on the dspace object. As you say above, single Context. The HashMap is eligible for garbage collection once you no longer use that Context. So you will only build up a large cache if you are using a SINGLE Context over a long period of time. (As a rough guide, for requests to the web interface there will be a single Context used for each request, for the duration of that request). But Context's that are held open for a very long time have the potential to run into all sorts of other issues - dropping of idle database connections, lock contentions/waits, excessive database resource usage. And any error in that long run could cause the entire process to be rolled back - which may not be the best course of action. If you are running anything that takes an excessive amount of time, it's better to really consider if there are better ways you can be using (/releasing) contexts, before jumping to clear the cache. G This e-mail is confidential and should not be used by anyone who is not the original intended recipient. BioMed Central Limited does not accept liability for any statements made which are clearly the sender's own and not expressly made on behalf of BioMed Central Limited. No contracts may be concluded on behalf of BioMed Central Limited by means of e-mail communication. BioMed Central Limited Registered in England and Wales with registered number 3680030 Registered Office Middlesex House, 34-42 Cleveland Street, London W1T 4LB This email has been scanned by Postini. For more information please visit http://www.postini.com - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] FW: Java heap space error when trying to update a Collection
We've noticed issues with heap size as well, specifically when using the wildcard policy tool, the media filter manager, and some in-house tools that iterate over dspace objects using a single Context. When dspace objects like Item or Bitstream are instantiated, they place a reference to itself in a hash map attached to the context. So over time that map will get large. We modified the tools to clear the context "cache" when they were done operating on the dspace object. IIRC, methods to manage the cache weren't exposed until DSpace 1.5. I'm not sure if you are seeing the same issue or not, but this might be another thing to look at. HTH, Elliot Thornton, Susan M. (LARC-B702)[NCI INFORMATION SYSTEMS] wrote: > > From: François Parmentier [mailto:[EMAIL PROTECTED] > Sent: Monday, September 29, 2008 3:53 AM > To: Thornton, Susan M. (LARC-B702)[NCI INFORMATION SYSTEMS] > Subject: Re: [Dspace-tech] Java heap space error when trying to update a > Collection > > > > Unfortunately, I agree with Susan. > We have a 300K items collection that we cannot upgrade (only changing the > description is not possible). > It happens only on THAT collection (and not on the other ones in the same > DSpace 1.4.2). > I did not look far inside, but it seems that DSpace loads all the > documents of the collection to be updated during the update! > > I'm not on that problem right now. > -- > François PARMENTIER / INIST-CNRS > OpenSIGLE > > On Fri, Sep 26, 2008 at 11:13 PM, Thornton, Susan M. (LARC-B702)[NCI > INFORMATION SYSTEMS] <[EMAIL PROTECTED]> wrote: > > Which memory settings might be too low? > > > > I'm thinking it has to be related to the increased number of records in > our database since this is the first time I've ever seen the error and > because it doesn't happen in our test instance (where our source code > mirrors our production system). I can't imagine though, what could be > happening in edit-communities where all we are trying to do is update one > row in the collection table?? > > > > > > From: Mark Diggory [mailto:[EMAIL PROTECTED] On Behalf Of Diggory Mark > Sent: Friday, September 26, 2008 5:09 PM > To: Thornton, Susan M. (LARC-B702)[NCI INFORMATION SYSTEMS] > Cc: dspace-tech@lists.sourceforge.net; Smail, James W. (LARC-B702)[NCI > INFORMATION SYSTEMS] > Subject: Re: [Dspace-tech] Java heap space error when trying to update a > Collection > > > > This can be caused by something else happening on the server. Basically > your instance has used up its JVM heap space and any new calls into the > system results in an OutOf Memory error. Of you can restart and try the > task again, you may find that the issue is gone, if its still there, then > you have an issue that being cause by your request more specifically. > Monitor/Search your dspace logs and see if you have more occuarnces of > this than this one case. It may either mean: > > > > 1.) your memory settings are too low. > > > > 2.) something is causing a memory leak, i.e. not releasing its resources > or falling into infinite loops. > > > > -Mark > > > > On Sep 26, 2008, at 1:54 PM, Thornton, Susan M. (LARC-B702)[NCI > INFORMATION SYSTEMS] wrote: > > > > I am getting an error while trying to update a Collection (collection name > and description) - see below from dspace.log. This is the first time I've > seen this error. The strange thing is that it doesn't happen in our Test > instance - but our Test instance only has 101 Items in it. We've recently > been adding lots of new items to our repository and our Item count is now > at 90,490. This is kind of scary, since we have thousands and thousands > more items we are planning on adding to DSpace in the next few months. > > We are currently running 1.4.2 and are planning to implement 1.5 soon. > > Can anyone help with this error? > > Thanks, > > Sue > > > > > > 2008-09-26 16:40:21,210 INFO org.dspace.eperson.PasswordAuthentication @ > anonymous:session_id=3DF9C128D08B43905296703AB6278699:ip_addr=198.119.152.109:authenticate:attempting > password auth of [EMAIL PROTECTED] > > 2008-09-26 16:40:21,240 INFO org.dspace.eperson.PasswordAuthentication > @[EMAIL > PROTECTED]:session_id=3DF9C128D08B43905296703AB6278699:ip_addr=198.119.152.109:authenticate:type=PasswordAuthentication > > 2008-09-26 16:40:21,270 INFO org.dspace.app.webui.servlet.PasswordServlet > @[EMAIL > PROTECTED]:session_id=3DF9C128D08B43905296703AB6278699:ip_addr=198.119.152.109:login:type=explicit > > 2008-09-26 16:40:21,288 INFO org.dspace.app.webui.servlet.MyDSpaceServlet > @[EMAIL > PROTECTED]:session_id=3DF9C128D08B43905296703AB6278699:ip_addr=198.119.152.109:view_mydspace: > > 2008-09-26 16:40:24,141 INFO > org.dspace.app.webui.servlet.CommunityListServlet > @[EMAIL > PROTECTED]:session_id=3DF9C128D08B43905296703AB6278699:ip_addr=198.119.152.109:view_community_list: > > 2008-09-26 16:40:28,218 INFO org.dspace.app.webui.servl
[Dspace-tech] FW: Java heap space error when trying to update a Collection
From: François Parmentier [mailto:[EMAIL PROTECTED] Sent: Monday, September 29, 2008 3:53 AM To: Thornton, Susan M. (LARC-B702)[NCI INFORMATION SYSTEMS] Subject: Re: [Dspace-tech] Java heap space error when trying to update a Collection Unfortunately, I agree with Susan. We have a 300K items collection that we cannot upgrade (only changing the description is not possible). It happens only on THAT collection (and not on the other ones in the same DSpace 1.4.2). I did not look far inside, but it seems that DSpace loads all the documents of the collection to be updated during the update! I'm not on that problem right now. -- François PARMENTIER / INIST-CNRS OpenSIGLE On Fri, Sep 26, 2008 at 11:13 PM, Thornton, Susan M. (LARC-B702)[NCI INFORMATION SYSTEMS] <[EMAIL PROTECTED]> wrote: Which memory settings might be too low? I'm thinking it has to be related to the increased number of records in our database since this is the first time I've ever seen the error and because it doesn't happen in our test instance (where our source code mirrors our production system). I can't imagine though, what could be happening in edit-communities where all we are trying to do is update one row in the collection table?? From: Mark Diggory [mailto:[EMAIL PROTECTED] On Behalf Of Diggory Mark Sent: Friday, September 26, 2008 5:09 PM To: Thornton, Susan M. (LARC-B702)[NCI INFORMATION SYSTEMS] Cc: dspace-tech@lists.sourceforge.net; Smail, James W. (LARC-B702)[NCI INFORMATION SYSTEMS] Subject: Re: [Dspace-tech] Java heap space error when trying to update a Collection This can be caused by something else happening on the server. Basically your instance has used up its JVM heap space and any new calls into the system results in an OutOf Memory error. Of you can restart and try the task again, you may find that the issue is gone, if its still there, then you have an issue that being cause by your request more specifically. Monitor/Search your dspace logs and see if you have more occuarnces of this than this one case. It may either mean: 1.) your memory settings are too low. 2.) something is causing a memory leak, i.e. not releasing its resources or falling into infinite loops. -Mark On Sep 26, 2008, at 1:54 PM, Thornton, Susan M. (LARC-B702)[NCI INFORMATION SYSTEMS] wrote: I am getting an error while trying to update a Collection (collection name and description) - see below from dspace.log. This is the first time I've seen this error. The strange thing is that it doesn't happen in our Test instance - but our Test instance only has 101 Items in it. We've recently been adding lots of new items to our repository and our Item count is now at 90,490. This is kind of scary, since we have thousands and thousands more items we are planning on adding to DSpace in the next few months. We are currently running 1.4.2 and are planning to implement 1.5 soon. Can anyone help with this error? Thanks, Sue 2008-09-26 16:40:21,210 INFO org.dspace.eperson.PasswordAuthentication @ anonymous:session_id=3DF9C128D08B43905296703AB6278699:ip_addr=198.119.152.109:authenticate:attempting password auth of [EMAIL PROTECTED] 2008-09-26 16:40:21,240 INFO org.dspace.eperson.PasswordAuthentication @[EMAIL PROTECTED]:session_id=3DF9C128D08B43905296703AB6278699:ip_addr=198.119.152.109:authenticate:type=PasswordAuthentication 2008-09-26 16:40:21,270 INFO org.dspace.app.webui.servlet.PasswordServlet @[EMAIL PROTECTED]:session_id=3DF9C128D08B43905296703AB6278699:ip_addr=198.119.152.109:login:type=explicit 2008-09-26 16:40:21,288 INFO org.dspace.app.webui.servlet.MyDSpaceServlet @[EMAIL PROTECTED]:session_id=3DF9C128D08B43905296703AB6278699:ip_addr=198.119.152.109:view_mydspace: 2008-09-26 16:40:24,141 INFO org.dspace.app.webui.servlet.CommunityListServlet @[EMAIL PROTECTED]:session_id=3DF9C128D08B43905296703AB6278699:ip_addr=198.119.152.109:view_community_list: 2008-09-26 16:40:28,218 INFO org.dspace.app.webui.servlet.DSpaceServlet @[EMAIL PROTECTED]:session_id=3DF9C128D08B43905296703AB6278699:ip_addr=198.119.152.109:view_collection:collection_id=35 2008-09-26 16:43:07,240 WARN org.dspace.app.webui.servlet.InternalErrorServlet @ :session_id=3DF9C128D08B43905296703AB6278699:internal_error:-- URL Was:http://ldr.larc.nasa.gov/dspace/jsp/tools/edit-communities -- Method: POST -- Parameters were: -- license: "" -- copyright_text: "" -- provenance_description: "" -- action: "9" -- submit: "Update" -- create: "false" -- collection_id: "35" -- short_description: "" -- side_bar_text: "" -- introductory_text: "The CoreAero Collection contains documents dating from the late 1800's to 1958 including documents from National Advisory Committee for Aeronautics (NACA), other government agencies, both foreign and domestic, academic institutions, and corporations." -- community_id: "34" -- name: "Langley Core Aerospace Documents
