[jira] [Commented] (SOLR-6675) Solr webapp deployment is very slow with jmx/ in solrconfig.xml
[ https://issues.apache.org/jira/browse/SOLR-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14288613#comment-14288613 ] Forest Soup commented on SOLR-6675: --- We agree it's the suggester part. Thanks! Solr webapp deployment is very slow with jmx/ in solrconfig.xml - Key: SOLR-6675 URL: https://issues.apache.org/jira/browse/SOLR-6675 Project: Solr Issue Type: Bug Components: SolrCloud Affects Versions: 4.7 Environment: Linux Redhat 64bit Reporter: Forest Soup Priority: Critical Labels: performance Attachments: 1014.zip, callstack.png We have a SolrCloud with Solr version 4.7 with Tomcat 7. And our solr index(cores) are big(50~100G) each core. When we start up tomcat, the solr webapp deployment is very slow. From tomcat's catalina log, every time it takes about 10 minutes to get deployed. After we analyzing java core dump, we notice it's because the loading process cannot finish until the MBean calculation for large index is done. So we tried to remove the jmx/ from solrconfig.xml, after that, the loading of solr webapp only take about 1 minute. So we can sure the MBean calculation for large index is the root cause. Could you please point me if there is any async way to do statistic monitoring without jmx/ in solrconfig.xml, or let it do calculation after the deployment? Thanks! The callstack.png file in the attachment is the call stack of the long blocking thread which is doing statistics calculation. The catalina log of tomcat: INFO: Starting Servlet Engine: Apache Tomcat/7.0.54 Oct 13, 2014 2:00:29 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive /opt/ibm/solrsearch/tomcat/webapps/solr.war Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deployment of web application archive /opt/ibm/solrsearch/tomcat/webapps/solr.war has finished in 594,325 ms Time taken for solr app Deployment is about 10 minutes --- Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/manager Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/manager has finished in 2,035 ms Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/examples Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/examples has finished in 1,789 ms Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/docs Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/docs has finished in 1,037 ms Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/ROOT Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/ROOT has finished in 948 ms Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/host-manager Oct 13, 2014 2:10:30 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/host-manager has finished in 951 ms Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [http-bio-8080] Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [ajp-bio-8009] Oct 13, 2014 2:10:31 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 601506 ms -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6675) Solr webapp deployment is very slow with jmx/ in solrconfig.xml
[ https://issues.apache.org/jira/browse/SOLR-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14288614#comment-14288614 ] Forest Soup commented on SOLR-6675: --- We agree it's the suggester part. Thanks! Solr webapp deployment is very slow with jmx/ in solrconfig.xml - Key: SOLR-6675 URL: https://issues.apache.org/jira/browse/SOLR-6675 Project: Solr Issue Type: Bug Components: SolrCloud Affects Versions: 4.7 Environment: Linux Redhat 64bit Reporter: Forest Soup Priority: Critical Labels: performance Attachments: 1014.zip, callstack.png We have a SolrCloud with Solr version 4.7 with Tomcat 7. And our solr index(cores) are big(50~100G) each core. When we start up tomcat, the solr webapp deployment is very slow. From tomcat's catalina log, every time it takes about 10 minutes to get deployed. After we analyzing java core dump, we notice it's because the loading process cannot finish until the MBean calculation for large index is done. So we tried to remove the jmx/ from solrconfig.xml, after that, the loading of solr webapp only take about 1 minute. So we can sure the MBean calculation for large index is the root cause. Could you please point me if there is any async way to do statistic monitoring without jmx/ in solrconfig.xml, or let it do calculation after the deployment? Thanks! The callstack.png file in the attachment is the call stack of the long blocking thread which is doing statistics calculation. The catalina log of tomcat: INFO: Starting Servlet Engine: Apache Tomcat/7.0.54 Oct 13, 2014 2:00:29 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive /opt/ibm/solrsearch/tomcat/webapps/solr.war Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deployment of web application archive /opt/ibm/solrsearch/tomcat/webapps/solr.war has finished in 594,325 ms Time taken for solr app Deployment is about 10 minutes --- Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/manager Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/manager has finished in 2,035 ms Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/examples Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/examples has finished in 1,789 ms Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/docs Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/docs has finished in 1,037 ms Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/ROOT Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/ROOT has finished in 948 ms Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/host-manager Oct 13, 2014 2:10:30 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/host-manager has finished in 951 ms Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [http-bio-8080] Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [ajp-bio-8009] Oct 13, 2014 2:10:31 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 601506 ms -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6675) Solr webapp deployment is very slow with jmx/ in solrconfig.xml
[ https://issues.apache.org/jira/browse/SOLR-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14282424#comment-14282424 ] Yuting Woo commented on SOLR-6675: -- I found this solution: disable the search component https://github.com/Heliosearch/heliosearch/commit/b43988585f948965a5760bf60314fc76b1c099ff and the document from solr wiki: https://wiki.apache.org/solr/SolrPerformanceProblems#How_much_heap_space_do_I_need.3F Solr webapp deployment is very slow with jmx/ in solrconfig.xml - Key: SOLR-6675 URL: https://issues.apache.org/jira/browse/SOLR-6675 Project: Solr Issue Type: Bug Components: SolrCloud Affects Versions: 4.7 Environment: Linux Redhat 64bit Reporter: Forest Soup Priority: Critical Labels: performance Attachments: 1014.zip, callstack.png We have a SolrCloud with Solr version 4.7 with Tomcat 7. And our solr index(cores) are big(50~100G) each core. When we start up tomcat, the solr webapp deployment is very slow. From tomcat's catalina log, every time it takes about 10 minutes to get deployed. After we analyzing java core dump, we notice it's because the loading process cannot finish until the MBean calculation for large index is done. So we tried to remove the jmx/ from solrconfig.xml, after that, the loading of solr webapp only take about 1 minute. So we can sure the MBean calculation for large index is the root cause. Could you please point me if there is any async way to do statistic monitoring without jmx/ in solrconfig.xml, or let it do calculation after the deployment? Thanks! The callstack.png file in the attachment is the call stack of the long blocking thread which is doing statistics calculation. The catalina log of tomcat: INFO: Starting Servlet Engine: Apache Tomcat/7.0.54 Oct 13, 2014 2:00:29 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive /opt/ibm/solrsearch/tomcat/webapps/solr.war Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deployment of web application archive /opt/ibm/solrsearch/tomcat/webapps/solr.war has finished in 594,325 ms Time taken for solr app Deployment is about 10 minutes --- Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/manager Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/manager has finished in 2,035 ms Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/examples Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/examples has finished in 1,789 ms Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/docs Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/docs has finished in 1,037 ms Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/ROOT Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/ROOT has finished in 948 ms Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/host-manager Oct 13, 2014 2:10:30 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/host-manager has finished in 951 ms Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [http-bio-8080] Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [ajp-bio-8009] Oct 13, 2014 2:10:31 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 601506 ms -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6675) Solr webapp deployment is very slow with jmx/ in solrconfig.xml
[ https://issues.apache.org/jira/browse/SOLR-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14246497#comment-14246497 ] Forest Soup commented on SOLR-6675: --- Looks like thread searcherExecutor-5-thread-1 and searcherExecutor-6-thread-1 blocking the coreLoadExecutor-4-thread-1 and coreLoadExecutor-4-thread-2. And searcherExecutor-5-thread-1 and searcherExecutor-6-thread-1 are like suggester code. [~hossman] Could you please help to make sure? Thanks! Solr webapp deployment is very slow with jmx/ in solrconfig.xml - Key: SOLR-6675 URL: https://issues.apache.org/jira/browse/SOLR-6675 Project: Solr Issue Type: Bug Components: SolrCloud Affects Versions: 4.7 Environment: Linux Redhat 64bit Reporter: Forest Soup Priority: Critical Labels: performance Attachments: 1014.zip, callstack.png We have a SolrCloud with Solr version 4.7 with Tomcat 7. And our solr index(cores) are big(50~100G) each core. When we start up tomcat, the solr webapp deployment is very slow. From tomcat's catalina log, every time it takes about 10 minutes to get deployed. After we analyzing java core dump, we notice it's because the loading process cannot finish until the MBean calculation for large index is done. So we tried to remove the jmx/ from solrconfig.xml, after that, the loading of solr webapp only take about 1 minute. So we can sure the MBean calculation for large index is the root cause. Could you please point me if there is any async way to do statistic monitoring without jmx/ in solrconfig.xml, or let it do calculation after the deployment? Thanks! The callstack.png file in the attachment is the call stack of the long blocking thread which is doing statistics calculation. The catalina log of tomcat: INFO: Starting Servlet Engine: Apache Tomcat/7.0.54 Oct 13, 2014 2:00:29 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive /opt/ibm/solrsearch/tomcat/webapps/solr.war Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deployment of web application archive /opt/ibm/solrsearch/tomcat/webapps/solr.war has finished in 594,325 ms Time taken for solr app Deployment is about 10 minutes --- Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/manager Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/manager has finished in 2,035 ms Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/examples Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/examples has finished in 1,789 ms Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/docs Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/docs has finished in 1,037 ms Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/ROOT Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/ROOT has finished in 948 ms Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/host-manager Oct 13, 2014 2:10:30 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/host-manager has finished in 951 ms Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [http-bio-8080] Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [ajp-bio-8009] Oct 13, 2014 2:10:31 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 601506 ms -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6675) Solr webapp deployment is very slow with jmx/ in solrconfig.xml
[ https://issues.apache.org/jira/browse/SOLR-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234034#comment-14234034 ] Forest Soup commented on SOLR-6675: --- This is our JVM. And we have never tried the latest Solr 4.10.x. Any idea on how to resolve or workaround it? Thanks! java version 1.7.0 Java(TM) SE Runtime Environment (build pxa6470sr6-20131015_01(SR6)) IBM J9 VM (build 2.6, JRE 1.7.0 Linux amd64-64 Compressed References 20131013_170512 (JIT enabled, AOT enabled) J9VM - R26_Java726_SR6_20131013_1510_B170512 JIT - r11.b05_20131003_47443 GC - R26_Java726_SR6_20131013_1510_B170512_CMPRSS J9CL - 20131013_170512) JCL - 20131011_01 based on Oracle 7u45-b18 Solr webapp deployment is very slow with jmx/ in solrconfig.xml - Key: SOLR-6675 URL: https://issues.apache.org/jira/browse/SOLR-6675 Project: Solr Issue Type: Bug Components: SolrCloud Affects Versions: 4.7 Environment: Linux Redhat 64bit Reporter: Forest Soup Priority: Critical Labels: performance Attachments: callstack.png We have a SolrCloud with Solr version 4.7 with Tomcat 7. And our solr index(cores) are big(50~100G) each core. When we start up tomcat, the solr webapp deployment is very slow. From tomcat's catalina log, every time it takes about 10 minutes to get deployed. After we analyzing java core dump, we notice it's because the loading process cannot finish until the MBean calculation for large index is done. So we tried to remove the jmx/ from solrconfig.xml, after that, the loading of solr webapp only take about 1 minute. So we can sure the MBean calculation for large index is the root cause. Could you please point me if there is any async way to do statistic monitoring without jmx/ in solrconfig.xml, or let it do calculation after the deployment? Thanks! The callstack.png file in the attachment is the call stack of the long blocking thread which is doing statistics calculation. The catalina log of tomcat: INFO: Starting Servlet Engine: Apache Tomcat/7.0.54 Oct 13, 2014 2:00:29 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive /opt/ibm/solrsearch/tomcat/webapps/solr.war Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deployment of web application archive /opt/ibm/solrsearch/tomcat/webapps/solr.war has finished in 594,325 ms Time taken for solr app Deployment is about 10 minutes --- Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/manager Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/manager has finished in 2,035 ms Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/examples Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/examples has finished in 1,789 ms Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/docs Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/docs has finished in 1,037 ms Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/ROOT Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/ROOT has finished in 948 ms Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/host-manager Oct 13, 2014 2:10:30 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/host-manager has finished in 951 ms Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [http-bio-8080] Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [ajp-bio-8009] Oct 13, 2014 2:10:31 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 601506 ms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SOLR-6675) Solr webapp deployment is very slow with jmx/ in solrconfig.xml
[ https://issues.apache.org/jira/browse/SOLR-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234906#comment-14234906 ] Hoss Man commented on SOLR-6675: Forest: if there is an Object wait happening in getSearcher, that means some other thread is already warming up a searcher - a full thread dump of every thread would be helpful to see why/how. it looks like you are seeing the perfect storm of 3 problems: * getStatistics is being called when *registering* the MBean * getStatistics is blocking because it needs the searcher and the searcher is being warmed * something is making your searcher take 10 minutes to warm Solr webapp deployment is very slow with jmx/ in solrconfig.xml - Key: SOLR-6675 URL: https://issues.apache.org/jira/browse/SOLR-6675 Project: Solr Issue Type: Bug Components: SolrCloud Affects Versions: 4.7 Environment: Linux Redhat 64bit Reporter: Forest Soup Priority: Critical Labels: performance Attachments: callstack.png We have a SolrCloud with Solr version 4.7 with Tomcat 7. And our solr index(cores) are big(50~100G) each core. When we start up tomcat, the solr webapp deployment is very slow. From tomcat's catalina log, every time it takes about 10 minutes to get deployed. After we analyzing java core dump, we notice it's because the loading process cannot finish until the MBean calculation for large index is done. So we tried to remove the jmx/ from solrconfig.xml, after that, the loading of solr webapp only take about 1 minute. So we can sure the MBean calculation for large index is the root cause. Could you please point me if there is any async way to do statistic monitoring without jmx/ in solrconfig.xml, or let it do calculation after the deployment? Thanks! The callstack.png file in the attachment is the call stack of the long blocking thread which is doing statistics calculation. The catalina log of tomcat: INFO: Starting Servlet Engine: Apache Tomcat/7.0.54 Oct 13, 2014 2:00:29 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive /opt/ibm/solrsearch/tomcat/webapps/solr.war Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deployment of web application archive /opt/ibm/solrsearch/tomcat/webapps/solr.war has finished in 594,325 ms Time taken for solr app Deployment is about 10 minutes --- Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/manager Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/manager has finished in 2,035 ms Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/examples Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/examples has finished in 1,789 ms Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/docs Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/docs has finished in 1,037 ms Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/ROOT Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/ROOT has finished in 948 ms Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/host-manager Oct 13, 2014 2:10:30 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/host-manager has finished in 951 ms Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [http-bio-8080] Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [ajp-bio-8009] Oct 13, 2014 2:10:31 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 601506 ms -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail:
[jira] [Commented] (SOLR-6675) Solr webapp deployment is very slow with jmx/ in solrconfig.xml
[ https://issues.apache.org/jira/browse/SOLR-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234907#comment-14234907 ] Hoss Man commented on SOLR-6675: seems extremeley relevant to the JMXMonitorMap issues miller was looking into. Solr webapp deployment is very slow with jmx/ in solrconfig.xml - Key: SOLR-6675 URL: https://issues.apache.org/jira/browse/SOLR-6675 Project: Solr Issue Type: Bug Components: SolrCloud Affects Versions: 4.7 Environment: Linux Redhat 64bit Reporter: Forest Soup Priority: Critical Labels: performance Attachments: callstack.png We have a SolrCloud with Solr version 4.7 with Tomcat 7. And our solr index(cores) are big(50~100G) each core. When we start up tomcat, the solr webapp deployment is very slow. From tomcat's catalina log, every time it takes about 10 minutes to get deployed. After we analyzing java core dump, we notice it's because the loading process cannot finish until the MBean calculation for large index is done. So we tried to remove the jmx/ from solrconfig.xml, after that, the loading of solr webapp only take about 1 minute. So we can sure the MBean calculation for large index is the root cause. Could you please point me if there is any async way to do statistic monitoring without jmx/ in solrconfig.xml, or let it do calculation after the deployment? Thanks! The callstack.png file in the attachment is the call stack of the long blocking thread which is doing statistics calculation. The catalina log of tomcat: INFO: Starting Servlet Engine: Apache Tomcat/7.0.54 Oct 13, 2014 2:00:29 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive /opt/ibm/solrsearch/tomcat/webapps/solr.war Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deployment of web application archive /opt/ibm/solrsearch/tomcat/webapps/solr.war has finished in 594,325 ms Time taken for solr app Deployment is about 10 minutes --- Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/manager Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/manager has finished in 2,035 ms Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/examples Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/examples has finished in 1,789 ms Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/docs Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/docs has finished in 1,037 ms Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/ROOT Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/ROOT has finished in 948 ms Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/host-manager Oct 13, 2014 2:10:30 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/host-manager has finished in 951 ms Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [http-bio-8080] Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [ajp-bio-8009] Oct 13, 2014 2:10:31 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 601506 ms -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6675) Solr webapp deployment is very slow with jmx/ in solrconfig.xml
[ https://issues.apache.org/jira/browse/SOLR-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14232555#comment-14232555 ] Otis Gospodnetic commented on SOLR-6675: I've never heard or seen this before. Have you tried latest Solr 4.10.x? Which JVM is this on? Solr webapp deployment is very slow with jmx/ in solrconfig.xml - Key: SOLR-6675 URL: https://issues.apache.org/jira/browse/SOLR-6675 Project: Solr Issue Type: Bug Components: SolrCloud Affects Versions: 4.7 Environment: Linux Redhat 64bit Reporter: Forest Soup Priority: Critical Labels: performance Attachments: callstack.png We have a SolrCloud with Solr version 4.7 with Tomcat 7. And our solr index(cores) are big(50~100G) each core. When we start up tomcat, the solr webapp deployment is very slow. From tomcat's catalina log, every time it takes about 10 minutes to get deployed. After we analyzing java core dump, we notice it's because the loading process cannot finish until the MBean calculation for large index is done. So we tried to remove the jmx/ from solrconfig.xml, after that, the loading of solr webapp only take about 1 minute. So we can sure the MBean calculation for large index is the root cause. Could you please point me if there is any async way to do statistic monitoring without jmx/ in solrconfig.xml, or let it do calculation after the deployment? Thanks! The callstack.png file in the attachment is the call stack of the long blocking thread which is doing statistics calculation. The catalina log of tomcat: INFO: Starting Servlet Engine: Apache Tomcat/7.0.54 Oct 13, 2014 2:00:29 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive /opt/ibm/solrsearch/tomcat/webapps/solr.war Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deployment of web application archive /opt/ibm/solrsearch/tomcat/webapps/solr.war has finished in 594,325 ms Time taken for solr app Deployment is about 10 minutes --- Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/manager Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/manager has finished in 2,035 ms Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/examples Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/examples has finished in 1,789 ms Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/docs Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/docs has finished in 1,037 ms Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/ROOT Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/ROOT has finished in 948 ms Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/host-manager Oct 13, 2014 2:10:30 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/host-manager has finished in 951 ms Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [http-bio-8080] Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [ajp-bio-8009] Oct 13, 2014 2:10:31 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 601506 ms -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6675) Solr webapp deployment is very slow with jmx/ in solrconfig.xml
[ https://issues.apache.org/jira/browse/SOLR-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14189798#comment-14189798 ] Forest Soup commented on SOLR-6675: --- The catalina log of tomcat: INFO: Starting Servlet Engine: Apache Tomcat/7.0.54 Oct 13, 2014 2:00:29 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive /opt/ibm/solrsearch/tomcat/webapps/solr.war Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deployment of web application archive /opt/ibm/solrsearch/tomcat/webapps/solr.war has finished in 594,325 ms Time taken for solr app Deployment is about 10 minutes --- Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/manager Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/manager has finished in 2,035 ms Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/examples Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/examples has finished in 1,789 ms Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/docs Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/docs has finished in 1,037 ms Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/ROOT Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/ROOT has finished in 948 ms Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/host-manager Oct 13, 2014 2:10:30 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/host-manager has finished in 951 ms Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [http-bio-8080] Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [ajp-bio-8009] Oct 13, 2014 2:10:31 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 601506 ms Solr webapp deployment is very slow with jmx/ in solrconfig.xml - Key: SOLR-6675 URL: https://issues.apache.org/jira/browse/SOLR-6675 Project: Solr Issue Type: Bug Components: SolrCloud Affects Versions: 4.7 Environment: Linux Redhat 64bit Reporter: Forest Soup Priority: Critical Labels: performance We have a SolrCloud with Solr version 4.7 with Tomcat 7. And our solr index(cores) are big(50~100G) each core. When we start up tomcat, the solr webapp deployment is very slow. From tomcat's catalina log, every time it takes about 10 minutes to get deployed. After we analyzing java core dump, we notice it's because the loading process cannot finish until the MBean calculation for large index is done. So we tried to remove the jmx/ from solrconfig.xml, after that, the loading of solr webapp only take about 1 minute. So we can sure the MBean calculation for large index is the root cause. Could you please point me if there is any async way to do statistic monitoring without jmx/ in solrconfig.xml, or let it do calculation after the deployment? Thanks! -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
