I encounter this problem as well. On my development desktop Ubuntu 9.10
spikes for seemingly indefinite periods of time. I tolerate the 100% spikes
because since I'm at the machine, its quick enough to kill process and
restart, and all I want to do is keep developing. But it would be better to
resolve.
Other notes:
Its hard to kill tomcat when its fully spiked.
i.e. sudo service tomcat6 stop doesn't stop tomcat.. I have to hunt and kill
it a few times for it to go down.
Here's my logs, and then I have a potential remedy at the end of this email.
After doing a flurry of activity, I can recreate the spike. 1 CPU at 100%
=============
tail -f /var/log/syslog
Mar 29 12:55:08 ACK5859s3 jsvc.exec[29783]: - Loading provided config file:
/dspace/config/dspace.cfg
Mar 29 12:55:08 ACK5859s3 jsvc.exec[29783]: - Using dspace provided log
configuration (log.init.config)#012- Loading:
/dspace/config/log4j.properties
Mar 29 12:55:08 ACK5859s3 jsvc.exec[29783]: Mar 29, 2010 11:55:08 AM
org.apache.catalina.startup.HostConfig checkResources#012INFO: Reloading
context [/xmlui]
Mar 29 12:55:09 ACK5859s3 jsvc.exec[29783]: log4j:WARN No appenders could be
found for logger (org.dspace.servicemanager.DSpaceKernelInit).#012log4j:WARN
Please initialize the log4j system properly.
Mar 29 12:55:10 ACK5859s3 jsvc.exec[29783]: INFO: Loading provided config
file: /dspace/config/dspace.cfg
Mar 29 12:55:21 ACK5859s3 jsvc.exec[29783]: Exception in thread
"ContainerBackgroundProcessor[StandardEngine[Catalina]]"
Mar 29 12:55:21 ACK5859s3 jsvc.exec[29783]: java.lang.OutOfMemoryError:
PermGen space
....
....
Mar 30 12:12:09 ACK5859s3 jsvc.exec[10801]: Mar 30, 2010 11:12:09 AM
org.apache.solr.core.SolrCore execute#012INFO: [statistics] webapp=null
path=null params={q=static+newSearcher+warming+query+from+solrconfig.xml}
hits=0 status=0 QTime=1 #012Mar 30, 2010 11:12:09 AM
org.apache.solr.core.QuerySenderListener newSearcher#012INFO:
QuerySenderListener done.
Mar 30 12:12:09 ACK5859s3 jsvc.exec[10801]: Mar 30, 2010 11:12:09 AM
org.apache.solr.core.SolrCore registerSearcher#012INFO: [statistics]
Registered new searcher searc...@59c4fcd4 main
Mar 30 12:12:09 ACK5859s3 jsvc.exec[10801]: Mar 30, 2010 11:12:09 AM
org.apache.solr.search.SolrIndexSearcher close#012INFO: Closing
searc...@25f39756main#012#011filtercache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}#012#011queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=6,evictions=0,size=6,warmupTime=97,cumulative_lookups=2,cumulative_hits=1,cumulative_hitratio=0.50,cumulative_inserts=1,cumulative_evictions=0}#012#011documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 30 12:12:19 ACK5859s3 jsvc.exec[10801]: Mar 30, 2010 11:12:19 AM
org.apache.coyote.http11.Http11AprProcessor process#012SEVERE: Error
processing request#012java.lang.OutOfMemoryError: PermGen space
Mar 30 12:17:01 ACK5859s3 CRON[9952]: (root) CMD ( cd / && run-parts
--report /etc/cron.hourly)
Mar 30 12:18:50 ACK5859s3 jsvc.exec[10801]: Exception in thread
"http-8080-1"
Mar 30 12:18:50 ACK5859s3 jsvc.exec[10801]: java.lang.OutOfMemoryError:
PermGen space
Mar 30 12:18:55 ACK5859s3 jsvc.exec[10801]: Exception in thread
"http-8080-6" java.lang.OutOfMemoryError: PermGen space
===================
tail -f dspace.log
2010-03-30 11:14:40,559 DEBUG org.dspace.app.webui.servlet.DSpaceServlet @
[email protected]:
session_id=21BB034E1977D0D9B10A6628FF5CF698:ip_addr=128.146.175.194:http_request:--
URL Was\colon; http\colon;//ack5859s3.lib.ohio-state.edu
\colon;8080/jspui/mydspace
-- Method\colon; POST
-- Parameters were\colon;
-- workflow_id\colon; "9757"
-- step\colon; "3"
-- submit_edit\colon; "Edit Metadata"
2010-03-30 11:14:42,122 DEBUG org.dspace.storage.rdbms.DatabaseManager @
Running query "SELECT * FROM group2groupcache WHERE child_id= ? OR
child_id= ? OR child_id= ? " with parameters: 0,1,1736
2010-03-30 11:14:42,123 DEBUG org.dspace.storage.rdbms.DatabaseManager @
Running query "select * from handle where handle = ? " with parameters:
1811/5839
2010-03-30 11:14:42,123 DEBUG org.dspace.storage.rdbms.DatabaseManager @
Running query "select * from community where community_id = ? " with
parameters: 35
2010-03-30 11:14:42,124 DEBUG org.dspace.content.Community @
[email protected]:
session_id=35AE8630F3CC08B9FC3522B9276B093A:ip_addr=127.0.1.1:
find_community:community_id=35
2010-03-30 11:14:42,124 DEBUG org.dspace.storage.rdbms.DatabaseManager @
Running query "SELECT handle FROM Handle WHERE resource_type_id = ? AND
resource_id = ?" with parameters: 4,35
2010-03-30 11:14:42,124 DEBUG org.dspace.handle.HandleManager @ Resolved
handle 1811/5839 to community 35
2010-03-30 11:14:42,127 DEBUG org.dspace.app.xmlui.cocoon.DSpaceLocaleAction
@ Found locale: en
2010-03-30 11:14:42,128 DEBUG org.dspace.app.xmlui.cocoon.AspectGenerator @
aspectgenerator path: cocoon://DRI/1/handle/1811/5839
2010-03-30 11:18:43,229 DEBUG org.dspace.storage.rdbms.DatabaseManager @
Running query "select * from workflowitem where workflow_id = ? " with
parameters: 9757
2010-03-30 11:18:44,030 DEBUG org.dspace.workflow.WorkflowItem @
[email protected]:
session_id=21BB034E1977D0D9B10A6628FF5CF698:ip_addr=128.146.175.194:
find_workflow_item:workflow_id=9757
2010-03-30 11:18:48,044 DEBUG org.dspace.storage.rdbms.DatabaseManager @
Running query "select * from item where item_id = ? " with parameters:
46388
2010-03-30 11:18:48,045 DEBUG org.dspace.content.Item @ [email protected]:
session_id=21BB034E1977D0D9B10A6628FF5CF698:ip_addr=128.146.175.194:
find_item:item_id=46388
2010-03-30 11:18:48,045 DEBUG org.dspace.storage.rdbms.DatabaseManager @
Running query "SELECT * FROM MetadataValue WHERE item_id= ? ORDER BY
metadata_field_id, place" with parameters: 46388
2010-03-30 11:18:48,046 DEBUG org.dspace.storage.rdbms.DatabaseManager @
Running query "SELECT handle FROM Handle WHERE resource_type_id = ? AND
resource_id = ?" with parameters: 2,46388
2010-03-30 11:18:48,046 DEBUG org.dspace.storage.rdbms.DatabaseManager @
Running query "select * from collection where collection_id = ? " with
parameters: 1319
2010-03-30 11:18:48,046 DEBUG org.dspace.content.Collection @
[email protected]:
session_id=21BB034E1977D0D9B10A6628FF5CF698:ip_addr=128.146.175.194:
find_collection:collection_id=1319
2010-03-30 11:18:48,046 DEBUG org.dspace.storage.rdbms.DatabaseManager @
Running query "select * from epersongroup where eperson_group_id = ? " with
parameters: 1736
2010-03-30 11:18:48,047 DEBUG org.dspace.storage.rdbms.DatabaseManager @
Running query "select * from epersongroup where eperson_group_id = ? " with
parameters: 1734
2010-03-30 11:18:48,047 DEBUG org.dspace.storage.rdbms.DatabaseManager @
Running query "select * from epersongroup where eperson_group_id = ? " with
parameters: 1738
2010-03-30 11:18:48,047 DEBUG org.dspace.storage.rdbms.DatabaseManager @
Running query "SELECT handle FROM Handle WHERE resource_type_id = ? AND
resource_id = ?" with parameters: 3,1319
Notice the four minute time gap. I had created the spike, left it alone, and
ever so slowly, it would be able to commit something. So thats looking like
a resource issue.
===================
For reference, this is all postgres is saying...
pe...@ack5859s3:~$ sudo tail -f /var/log/postgresql/postgresql-8.4-main.log
2010-03-30 12:30:34 EDT LOG: unexpected EOF on client connection
2010-03-30 12:30:34 EDT LOG: unexpected EOF on client connection
2010-03-30 12:30:34 EDT LOG: unexpected EOF on client connection
2010-03-30 12:30:34 EDT LOG: unexpected EOF on client connection
2010-03-30 12:30:34 EDT LOG: unexpected EOF on client connection
2010-03-30 12:30:34 EDT LOG: unexpected EOF on client connection
### I usually notice this after a tomcat restart
===========================
===========================
So after noticing the PermGen error in the log. I've decided to raise its
limit.
PermGen is memory that is permanently allocated, so I'm assuming that
certain things that only need to warm up once, and take a while for the
initial start, and then work fine, such as deploying a webapp, or making a
solr query would like this extra space.
So I edited /etc/init.d/tomcat6
if [ -z "$JAVA_OPTS" ]; then
JAVA_OPTS="-Djava.awt.headless=true -Xmx1536M *-XX:MaxPermSize=128m*
"
fi
After that, I can recreate the 100% spike after a flurry of activity, mainly
only during the initial page load, as the webapp initializes or SOLR
starts/warms... but its faster at resolving it, and it becomes too hard for
me to recreate the 100% CPU usage for eternity.
To address the memory usage that you brought up. Sure 1.5GB is a lot of
memory. But I think XMLUI is somewhat expensive in that regard. And plus,
just imagine how much memory Google probably requires, and 1.5GB pales in
comparison to 100petabytes or whatever.
On Mon, Mar 29, 2010 at 12:11 PM, Fabien COMBERNOUS
<[email protected]>wrote:
> Thomas Zastrow wrote:
> > Dear all,
> >
> > we are new to DSpace and try to install it on an Ubuntu server with
> > PostgreSQL and Apache Tomcat.
> >
> > We followed the instructions in the manual and everything seems to
> > compile and deploy. But when we click on the "xmlui" application, the
> > Java process on the machine goes up to 1.6 GB (!) memory and 2 of the 4
> > processors are at 100%. The browser is waiting and waiting, but the page
> > is still blank. We have to kill the Tomcat process manually. The Tomcat
> > log files contain no error or other useful hints.
> >
> > We are thankful for any hints!
> >
> > Best wishes from Germany,
> >
> > Marie and Tom
> >
> >
> Hi from France my neighbor ,
>
> Here i'm also using dspace (1.6rc2) but with Debian Lenny. I'm using
> sun-java6-jre and all is fine. what is your jre ?
>
>
> --
> *Fabien COMBERNOUS*
> /unix system engineer/
> www.kezia.com <http://www.kezia.com/>
> *Tel: +33 (0) 467 992 986*
> Kezia Group
>
>
> ------------------------------------------------------------------------------
> Download Intel® Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> DSpace-tech mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
>
--
Peter Dietz
Systems Developer/Engineer
Ohio State University Libraries
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
DSpace-tech mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dspace-tech
