[ https://issues.apache.org/jira/browse/SOLR-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234906#comment-14234906 ]
Hoss Man commented on SOLR-6675: -------------------------------- Forest: if there is an Object "wait" happening in getSearcher, that means some other thread is already warming up a searcher - a full thread dump of every thread would be helpful to see why/how. it looks like you are seeing the perfect storm of 3 problems: * getStatistics is being called when *registering* the MBean * getStatistics is blocking because it needs the searcher and the searcher is being warmed * something is making your searcher take 10 minutes to warm > Solr webapp deployment is very slow with <jmx/> in solrconfig.xml > ----------------------------------------------------------------- > > Key: SOLR-6675 > URL: https://issues.apache.org/jira/browse/SOLR-6675 > Project: Solr > Issue Type: Bug > Components: SolrCloud > Affects Versions: 4.7 > Environment: Linux Redhat 64bit > Reporter: Forest Soup > Priority: Critical > Labels: performance > Attachments: callstack.png > > > We have a SolrCloud with Solr version 4.7 with Tomcat 7. And our solr > index(cores) are big(50~100G) each core. > When we start up tomcat, the solr webapp deployment is very slow. From > tomcat's catalina log, every time it takes about 10 minutes to get deployed. > After we analyzing java core dump, we notice it's because the loading process > cannot finish until the MBean calculation for large index is done. > > So we tried to remove the <jmx/> from solrconfig.xml, after that, the loading > of solr webapp only take about 1 minute. So we can sure the MBean calculation > for large index is the root cause. > Could you please point me if there is any async way to do statistic > monitoring without <jmx/> in solrconfig.xml, or let it do calculation after > the deployment? Thanks! > The callstack.png file in the attachment is the call stack of the long > blocking thread which is doing statistics calculation. > The catalina log of tomcat: > INFO: Starting Servlet Engine: Apache Tomcat/7.0.54 > Oct 13, 2014 2:00:29 AM org.apache.catalina.startup.HostConfig deployWAR > INFO: Deploying web application archive > /opt/ibm/solrsearch/tomcat/webapps/solr.war > Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployWAR > INFO: Deployment of web application archive > /opt/ibm/solrsearch/tomcat/webapps/solr.war has finished in 594,325 ms > <---------------- Time taken for solr app Deployment is about 10 minutes > ----------- > Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployDirectory > INFO: Deploying web application directory > /opt/ibm/solrsearch/tomcat/webapps/manager > Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory > INFO: Deployment of web application directory > /opt/ibm/solrsearch/tomcat/webapps/manager has finished in 2,035 ms > Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory > INFO: Deploying web application directory > /opt/ibm/solrsearch/tomcat/webapps/examples > Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory > INFO: Deployment of web application directory > /opt/ibm/solrsearch/tomcat/webapps/examples has finished in 1,789 ms > Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory > INFO: Deploying web application directory > /opt/ibm/solrsearch/tomcat/webapps/docs > Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory > INFO: Deployment of web application directory > /opt/ibm/solrsearch/tomcat/webapps/docs has finished in 1,037 ms > Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory > INFO: Deploying web application directory > /opt/ibm/solrsearch/tomcat/webapps/ROOT > Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory > INFO: Deployment of web application directory > /opt/ibm/solrsearch/tomcat/webapps/ROOT has finished in 948 ms > Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory > INFO: Deploying web application directory > /opt/ibm/solrsearch/tomcat/webapps/host-manager > Oct 13, 2014 2:10:30 AM org.apache.catalina.startup.HostConfig deployDirectory > INFO: Deployment of web application directory > /opt/ibm/solrsearch/tomcat/webapps/host-manager has finished in 951 ms > Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start > INFO: Starting ProtocolHandler ["http-bio-8080"] > Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start > INFO: Starting ProtocolHandler ["ajp-bio-8009"] > Oct 13, 2014 2:10:31 AM org.apache.catalina.startup.Catalina start > INFO: Server startup in 601506 ms -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org