RE: suspect SOLR query from D029 (SOLR master)
Krupansky [mailto:j...@basetechnology.com] Sent: Tuesday, June 03, 2014 6:53 AM To: solr-user@lucene.apache.org Subject: Re: suspect SOLR query from D029 (SOLR master) 11 * 11 or 121 query terms, which shouldn't be so bad. But... maybe the Lucene FST for your synonym list is huge. Someone with deeper Lucene knowledge would have to address that. -- Jack Krupansky -Original Message- From: Branham, Jeremy [HR] Sent: Tuesday, June 3, 2014 3:57 AM To: solr-user@lucene.apache.org Cc: Worley, Chris [HR] Subject: RE: suspect SOLR query from D029 (SOLR master) Evidently I didn't understand enough about the synonym filter. I'm not sure anyone would be able to determine the impact based on the example queries below. However I'm curious what the best practice is for synonyms. We have 179 lines of synonyms each with 2 - 6 synonyms per line [all expanded]. The query that hurt us had 11 terms in the search string that matched a synonym line with 11 values. How does this work internally? Does the query parser create 11^11 search terms that would be parsed? Should we be looking closer at stemming and tokenization rather than creating this list of synonyms? Just looking for some clarity, or any best practice for use cases that might call for synonyms. Thank you. Jeremy D. Branham Tel: **DOTNET -Original Message- From: Branham, Jeremy [HR] Sent: Monday, June 02, 2014 9:49 PM To: solr-user@lucene.apache.org Cc: Worley, Chris [HR] Subject: RE: suspect SOLR query from D029 (SOLR master) These are the typical queries we are using. I'm curious if any of these parameters could be causing issues when using synonyms. ?shards=myserver1.com:8080/svc/solr/wdsc,myserver1.com:8080/svc/solr/kmssort=score descq=(keyword:(this is a test) OR titleSearch:(this is a test) AND (doctype:(Device-Product)^1.1 OR (doctype:(LaunchPacks) OR doctype:(MenuMaps) OR doctype:(HowToSetupGuide) OR doctype:(ServiceAdvisory) OR doctype:(TroubleshootingGuide) OR doctype:(FAQ) OR doctype:(DeviceManualdefType=edismaxfq=device_id:(Motorola Moto X \(XT1056\))fq=-doctype:Simulator AND -doctype:Scenariobq=doctype:LaunchPacks^1.2 OR doctype:MenuMaps^1.2 OR doctype:HowToSetupGuide OR doctype:ServiceAdvisory^1.2 OR doctype:TroubleshootingGuide^1.4 OR doctype:FAQ^2.0 OR doctype:DeviceManual OR (doctype:Device-Product AND titleSearch:(Programming \-))^15 OR (doctype:Device-Product AND titleSearch:(Solution Matrix \-))^5 OR (doctype:Device-Product AND titleSearch:(Fact Sheet \-))^4 OR (doctype:Device-Product AND titleSearch:(User Guide \-))^3fl=id,title,device_id,url,doctype,titleSearch,scoremm=1start=0rows=10 ?sort=score desc,hits30day descfq=profile:YYfq=channel:ASPC_CustManaCMq=this is a testmm=0start=0rows=10 ?fq=device_id:(Motorola Moto X \(XT1056\))q=(keyword:(this is a test) OR titleSearch:(this is a test))start=0rows=10 ?sort=score descfq=doctype:ProductKnownIssuefq=(technology:3G 1900/800, LTE both) q=(text:Motorola Moto X (XT1056)^1000 OR text:[* TO *]) AND (titleSearch:(this is a test)^2 OR text:(this is a test)^1.5)start=0rows=10 ?sort=score descq=((titleSearch:(this is a test)^2 OR text:(this is a test)) AND ((device_id:Motorola Moto X \(XT1056\) AND category:4247)^20 OR (category:3676 AND profile:YY)))fq=doctype:FlowDocfq=(-voice_network:[* TO *] AND *:*) OR voice_network:3G 1900/800, LTEstart=0rows=10 ?fq=doctype:Scenarioq=device_id:Motorola Moto X (XT1056) AND (titleSearch:(this is a test)^2 OR text:(this is a test)^1.5)start=0rows=10 Jeremy D. Branham Tel: **DOTNET -Original Message- From: Branham, Jeremy [HR] Sent: Monday, June 02, 2014 8:31 PM To: solr-user@lucene.apache.org Subject: RE: suspect SOLR query from D029 (SOLR master) We found a problem with the synonym list, and suspect there was some sort of recursion causing the memory to be gobbled up until the JVM crashed. Is this expected behavior from complex synonyms? Or could this be due to the combination of complex synonyms and a bad query format? Jeremy D. Branham Tel: **DOTNET -Original Message- From: Branham, Jeremy [HR] Sent: Friday, May 30, 2014 4:43 PM To: solr-user@lucene.apache.org Subject: RE: suspect SOLR query from D029 (SOLR master) We've switched to CMS GC to see if there is any improvement. Looking at this use case, G1GC might have been a better option, but we are running JDK 1.6 http://blog.sematext.com/2013/06/24/g1-cms-java-garbage-collector/ Jeremy D. Branham Tel: **DOTNET -Original Message- From: Branham, Jeremy [HR] Sent: Friday, May 30, 2014 11:52 AM To: solr-user@lucene.apache.org Subject: FW: suspect SOLR query from D029 (SOLR master) We saw the file descriptors peak out and full GCs running causing DOS on our SOLR servers this morning. * Does this stack trace give enough information for some ideas? * solr-spec 4.5.1-SNAPSHOT * solr-impl 4.5.1-SNAPSHOT ${svnversion} - kx101435 - 2013-11-04 17:39:36 * lucene-spec 4.5.1 * lucene-impl 4.5.1
RE: suspect SOLR query from D029 (SOLR master)
Evidently I didn't understand enough about the synonym filter. I'm not sure anyone would be able to determine the impact based on the example queries below. However I'm curious what the best practice is for synonyms. We have 179 lines of synonyms each with 2 - 6 synonyms per line [all expanded]. The query that hurt us had 11 terms in the search string that matched a synonym line with 11 values. How does this work internally? Does the query parser create 11^11 search terms that would be parsed? Should we be looking closer at stemming and tokenization rather than creating this list of synonyms? Just looking for some clarity, or any best practice for use cases that might call for synonyms. Thank you. Jeremy D. Branham Tel: **DOTNET -Original Message- From: Branham, Jeremy [HR] Sent: Monday, June 02, 2014 9:49 PM To: solr-user@lucene.apache.org Cc: Worley, Chris [HR] Subject: RE: suspect SOLR query from D029 (SOLR master) These are the typical queries we are using. I'm curious if any of these parameters could be causing issues when using synonyms. ?shards=myserver1.com:8080/svc/solr/wdsc,myserver1.com:8080/svc/solr/kmssort=score descq=(keyword:(this is a test) OR titleSearch:(this is a test) AND (doctype:(Device-Product)^1.1 OR (doctype:(LaunchPacks) OR doctype:(MenuMaps) OR doctype:(HowToSetupGuide) OR doctype:(ServiceAdvisory) OR doctype:(TroubleshootingGuide) OR doctype:(FAQ) OR doctype:(DeviceManualdefType=edismaxfq=device_id:(Motorola Moto X \(XT1056\))fq=-doctype:Simulator AND -doctype:Scenariobq=doctype:LaunchPacks^1.2 OR doctype:MenuMaps^1.2 OR doctype:HowToSetupGuide OR doctype:ServiceAdvisory^1.2 OR doctype:TroubleshootingGuide^1.4 OR doctype:FAQ^2.0 OR doctype:DeviceManual OR (doctype:Device-Product AND titleSearch:(Programming \-))^15 OR (doctype:Device-Product AND titleSearch:(Solution Matrix \-))^5 OR (doctype:Device-Product AND titleSearch:(Fact Sheet \-))^4 OR (doctype:Device-Product AND titleSearch:(User Guide \-))^3fl=id,title,device_id,url,doctype,titleSearch,scoremm=1start=0rows=10 ?sort=score desc,hits30day descfq=profile:YYfq=channel:ASPC_CustManaCMq=this is a testmm=0start=0rows=10 ?fq=device_id:(Motorola Moto X \(XT1056\))q=(keyword:(this is a test) OR titleSearch:(this is a test))start=0rows=10 ?sort=score descfq=doctype:ProductKnownIssuefq=(technology:3G 1900/800, LTE both) q=(text:Motorola Moto X (XT1056)^1000 OR text:[* TO *]) AND (titleSearch:(this is a test)^2 OR text:(this is a test)^1.5)start=0rows=10 ?sort=score descq=((titleSearch:(this is a test)^2 OR text:(this is a test)) AND ((device_id:Motorola Moto X \(XT1056\) AND category:4247)^20 OR (category:3676 AND profile:YY)))fq=doctype:FlowDocfq=(-voice_network:[* TO *] AND *:*) OR voice_network:3G 1900/800, LTEstart=0rows=10 ?fq=doctype:Scenarioq=device_id:Motorola Moto X (XT1056) AND (titleSearch:(this is a test)^2 OR text:(this is a test)^1.5)start=0rows=10 Jeremy D. Branham Tel: **DOTNET -Original Message- From: Branham, Jeremy [HR] Sent: Monday, June 02, 2014 8:31 PM To: solr-user@lucene.apache.org Subject: RE: suspect SOLR query from D029 (SOLR master) We found a problem with the synonym list, and suspect there was some sort of recursion causing the memory to be gobbled up until the JVM crashed. Is this expected behavior from complex synonyms? Or could this be due to the combination of complex synonyms and a bad query format? Jeremy D. Branham Tel: **DOTNET -Original Message- From: Branham, Jeremy [HR] Sent: Friday, May 30, 2014 4:43 PM To: solr-user@lucene.apache.org Subject: RE: suspect SOLR query from D029 (SOLR master) We've switched to CMS GC to see if there is any improvement. Looking at this use case, G1GC might have been a better option, but we are running JDK 1.6 http://blog.sematext.com/2013/06/24/g1-cms-java-garbage-collector/ Jeremy D. Branham Tel: **DOTNET -Original Message- From: Branham, Jeremy [HR] Sent: Friday, May 30, 2014 11:52 AM To: solr-user@lucene.apache.org Subject: FW: suspect SOLR query from D029 (SOLR master) We saw the file descriptors peak out and full GCs running causing DOS on our SOLR servers this morning. * Does this stack trace give enough information for some ideas? * solr-spec 4.5.1-SNAPSHOT * solr-impl 4.5.1-SNAPSHOT ${svnversion} - kx101435 - 2013-11-04 17:39:36 * lucene-spec 4.5.1 * lucene-impl 4.5.1 1533280 - mark - 2013-10-17 21:40:03 We also saw some IO connection exceptions in the SOLR log - 2014-05-30 10:24:20,334 ERROR - SolrDispatchFilter - null:org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,334 ERROR - SolrCore - org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server
Re: suspect SOLR query from D029 (SOLR master)
11 * 11 or 121 query terms, which shouldn't be so bad. But... maybe the Lucene FST for your synonym list is huge. Someone with deeper Lucene knowledge would have to address that. -- Jack Krupansky -Original Message- From: Branham, Jeremy [HR] Sent: Tuesday, June 3, 2014 3:57 AM To: solr-user@lucene.apache.org Cc: Worley, Chris [HR] Subject: RE: suspect SOLR query from D029 (SOLR master) Evidently I didn't understand enough about the synonym filter. I'm not sure anyone would be able to determine the impact based on the example queries below. However I'm curious what the best practice is for synonyms. We have 179 lines of synonyms each with 2 - 6 synonyms per line [all expanded]. The query that hurt us had 11 terms in the search string that matched a synonym line with 11 values. How does this work internally? Does the query parser create 11^11 search terms that would be parsed? Should we be looking closer at stemming and tokenization rather than creating this list of synonyms? Just looking for some clarity, or any best practice for use cases that might call for synonyms. Thank you. Jeremy D. Branham Tel: **DOTNET -Original Message- From: Branham, Jeremy [HR] Sent: Monday, June 02, 2014 9:49 PM To: solr-user@lucene.apache.org Cc: Worley, Chris [HR] Subject: RE: suspect SOLR query from D029 (SOLR master) These are the typical queries we are using. I'm curious if any of these parameters could be causing issues when using synonyms. ?shards=myserver1.com:8080/svc/solr/wdsc,myserver1.com:8080/svc/solr/kmssort=score descq=(keyword:(this is a test) OR titleSearch:(this is a test) AND (doctype:(Device-Product)^1.1 OR (doctype:(LaunchPacks) OR doctype:(MenuMaps) OR doctype:(HowToSetupGuide) OR doctype:(ServiceAdvisory) OR doctype:(TroubleshootingGuide) OR doctype:(FAQ) OR doctype:(DeviceManualdefType=edismaxfq=device_id:(Motorola Moto X \(XT1056\))fq=-doctype:Simulator AND -doctype:Scenariobq=doctype:LaunchPacks^1.2 OR doctype:MenuMaps^1.2 OR doctype:HowToSetupGuide OR doctype:ServiceAdvisory^1.2 OR doctype:TroubleshootingGuide^1.4 OR doctype:FAQ^2.0 OR doctype:DeviceManual OR (doctype:Device-Product AND titleSearch:(Programming \-))^15 OR (doctype:Device-Product AND titleSearch:(Solution Matrix \-))^5 OR (doctype:Device-Product AND titleSearch:(Fact Sheet \-))^4 OR (doctype:Device-Product AND titleSearch:(User Guide \-))^3fl=id,title,device_id,url,doctype,titleSearch,scoremm=1start=0rows=10 ?sort=score desc,hits30day descfq=profile:YYfq=channel:ASPC_CustManaCMq=this is a testmm=0start=0rows=10 ?fq=device_id:(Motorola Moto X \(XT1056\))q=(keyword:(this is a test) OR titleSearch:(this is a test))start=0rows=10 ?sort=score descfq=doctype:ProductKnownIssuefq=(technology:3G 1900/800, LTE both) q=(text:Motorola Moto X (XT1056)^1000 OR text:[* TO *]) AND (titleSearch:(this is a test)^2 OR text:(this is a test)^1.5)start=0rows=10 ?sort=score descq=((titleSearch:(this is a test)^2 OR text:(this is a test)) AND ((device_id:Motorola Moto X \(XT1056\) AND category:4247)^20 OR (category:3676 AND profile:YY)))fq=doctype:FlowDocfq=(-voice_network:[* TO *] AND *:*) OR voice_network:3G 1900/800, LTEstart=0rows=10 ?fq=doctype:Scenarioq=device_id:Motorola Moto X (XT1056) AND (titleSearch:(this is a test)^2 OR text:(this is a test)^1.5)start=0rows=10 Jeremy D. Branham Tel: **DOTNET -Original Message- From: Branham, Jeremy [HR] Sent: Monday, June 02, 2014 8:31 PM To: solr-user@lucene.apache.org Subject: RE: suspect SOLR query from D029 (SOLR master) We found a problem with the synonym list, and suspect there was some sort of recursion causing the memory to be gobbled up until the JVM crashed. Is this expected behavior from complex synonyms? Or could this be due to the combination of complex synonyms and a bad query format? Jeremy D. Branham Tel: **DOTNET -Original Message- From: Branham, Jeremy [HR] Sent: Friday, May 30, 2014 4:43 PM To: solr-user@lucene.apache.org Subject: RE: suspect SOLR query from D029 (SOLR master) We've switched to CMS GC to see if there is any improvement. Looking at this use case, G1GC might have been a better option, but we are running JDK 1.6 http://blog.sematext.com/2013/06/24/g1-cms-java-garbage-collector/ Jeremy D. Branham Tel: **DOTNET -Original Message- From: Branham, Jeremy [HR] Sent: Friday, May 30, 2014 11:52 AM To: solr-user@lucene.apache.org Subject: FW: suspect SOLR query from D029 (SOLR master) We saw the file descriptors peak out and full GCs running causing DOS on our SOLR servers this morning. * Does this stack trace give enough information for some ideas? * solr-spec 4.5.1-SNAPSHOT * solr-impl 4.5.1-SNAPSHOT ${svnversion} - kx101435 - 2013-11-04 17:39:36 * lucene-spec 4.5.1 * lucene-impl 4.5.1 1533280 - mark - 2013-10-17 21:40:03 We also saw some IO connection exceptions in the SOLR log - 2014-05-30 10:24
RE: suspect SOLR query from D029 (SOLR master)
We found a problem with the synonym list, and suspect there was some sort of recursion causing the memory to be gobbled up until the JVM crashed. Is this expected behavior from complex synonyms? Or could this be due to the combination of complex synonyms and a bad query format? Jeremy D. Branham Tel: **DOTNET -Original Message- From: Branham, Jeremy [HR] Sent: Friday, May 30, 2014 4:43 PM To: solr-user@lucene.apache.org Subject: RE: suspect SOLR query from D029 (SOLR master) We've switched to CMS GC to see if there is any improvement. Looking at this use case, G1GC might have been a better option, but we are running JDK 1.6 http://blog.sematext.com/2013/06/24/g1-cms-java-garbage-collector/ Jeremy D. Branham Tel: **DOTNET -Original Message- From: Branham, Jeremy [HR] Sent: Friday, May 30, 2014 11:52 AM To: solr-user@lucene.apache.org Subject: FW: suspect SOLR query from D029 (SOLR master) We saw the file descriptors peak out and full GCs running causing DOS on our SOLR servers this morning. * Does this stack trace give enough information for some ideas? * solr-spec 4.5.1-SNAPSHOT * solr-impl 4.5.1-SNAPSHOT ${svnversion} - kx101435 - 2013-11-04 17:39:36 * lucene-spec 4.5.1 * lucene-impl 4.5.1 1533280 - mark - 2013-10-17 21:40:03 We also saw some IO connection exceptions in the SOLR log - 2014-05-30 10:24:20,334 ERROR - SolrDispatchFilter - null:org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,334 ERROR - SolrCore - org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://myserver.com:8080/svc/solr/wdsp 2014-05-30 10:24:20,333 ERROR - SolrCore - org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,335 ERROR - SolrCore - org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,336 ERROR - SolrDispatchFilter - null:org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,335 ERROR - SolrDispatchFilter - null:org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsp Jeremy D. Branham Tel: **DOTNET From: Worley, Chris [HR] Sent: Friday, May 30, 2014 11:38 AM To: Stoner, Susan [HR]; Branham, Jeremy [HR] Cc: Barrett, Kevin B [HR]; Aldrich, Daniel J [HR]; Duncan, Horace W [HR] Subject: suspect SOLR query from D029 (SOLR master) Looks like a suspect SOLR query was executed at approx. 8:54am this morning on the SOLR master server (D029) that caused the java GC to go through the roof. See below: 2014-05-30 08:54:17,092 ERROR - SolrDispatchFilter - null:java.lang.OutOfMemoryError: GC overhead limit exceeded at org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum$Frame.loadBlock(BlockTreeTermsReader.java:2377) at org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum.seekExact(BlockTreeTermsReader.java:1698) at org.apache.lucene.index.TermContext.build(TermContext.java:95) at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:166) at org.apache.lucene.search.BooleanQuery$BooleanWeight.init(BooleanQuery.java:183) at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384) at org.apache.lucene.search.BooleanQuery$BooleanWeight.init(BooleanQuery.java:183) at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384) at org.apache.lucene.search.BooleanQuery$BooleanWeight.init(BooleanQuery.java:183) at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384) at org.apache.lucene.search.BooleanQuery$BooleanWeight.init(BooleanQuery.java:183) at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384) at org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82) at org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:690) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297) at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1501) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1367
RE: suspect SOLR query from D029 (SOLR master)
These are the typical queries we are using. I'm curious if any of these parameters could be causing issues when using synonyms. ?shards=myserver1.com:8080/svc/solr/wdsc,myserver1.com:8080/svc/solr/kmssort=score descq=(keyword:(this is a test) OR titleSearch:(this is a test) AND (doctype:(Device-Product)^1.1 OR (doctype:(LaunchPacks) OR doctype:(MenuMaps) OR doctype:(HowToSetupGuide) OR doctype:(ServiceAdvisory) OR doctype:(TroubleshootingGuide) OR doctype:(FAQ) OR doctype:(DeviceManualdefType=edismaxfq=device_id:(Motorola Moto X \(XT1056\))fq=-doctype:Simulator AND -doctype:Scenariobq=doctype:LaunchPacks^1.2 OR doctype:MenuMaps^1.2 OR doctype:HowToSetupGuide OR doctype:ServiceAdvisory^1.2 OR doctype:TroubleshootingGuide^1.4 OR doctype:FAQ^2.0 OR doctype:DeviceManual OR (doctype:Device-Product AND titleSearch:(Programming \-))^15 OR (doctype:Device-Product AND titleSearch:(Solution Matrix \-))^5 OR (doctype:Device-Product AND titleSearch:(Fact Sheet \-))^4 OR (doctype:Device-Product AND titleSearch:(User Guide \-))^3fl=id,title,device_id,url,doctype,titleSearch,scoremm=1start=0rows=10 ?sort=score desc,hits30day descfq=profile:YYfq=channel:ASPC_CustManaCMq=this is a testmm=0start=0rows=10 ?fq=device_id:(Motorola Moto X \(XT1056\))q=(keyword:(this is a test) OR titleSearch:(this is a test))start=0rows=10 ?sort=score descfq=doctype:ProductKnownIssuefq=(technology:3G 1900/800, LTE both) q=(text:Motorola Moto X (XT1056)^1000 OR text:[* TO *]) AND (titleSearch:(this is a test)^2 OR text:(this is a test)^1.5)start=0rows=10 ?sort=score descq=((titleSearch:(this is a test)^2 OR text:(this is a test)) AND ((device_id:Motorola Moto X \(XT1056\) AND category:4247)^20 OR (category:3676 AND profile:YY)))fq=doctype:FlowDocfq=(-voice_network:[* TO *] AND *:*) OR voice_network:3G 1900/800, LTEstart=0rows=10 ?fq=doctype:Scenarioq=device_id:Motorola Moto X (XT1056) AND (titleSearch:(this is a test)^2 OR text:(this is a test)^1.5)start=0rows=10 Jeremy D. Branham Tel: **DOTNET -Original Message- From: Branham, Jeremy [HR] Sent: Monday, June 02, 2014 8:31 PM To: solr-user@lucene.apache.org Subject: RE: suspect SOLR query from D029 (SOLR master) We found a problem with the synonym list, and suspect there was some sort of recursion causing the memory to be gobbled up until the JVM crashed. Is this expected behavior from complex synonyms? Or could this be due to the combination of complex synonyms and a bad query format? Jeremy D. Branham Tel: **DOTNET -Original Message- From: Branham, Jeremy [HR] Sent: Friday, May 30, 2014 4:43 PM To: solr-user@lucene.apache.org Subject: RE: suspect SOLR query from D029 (SOLR master) We've switched to CMS GC to see if there is any improvement. Looking at this use case, G1GC might have been a better option, but we are running JDK 1.6 http://blog.sematext.com/2013/06/24/g1-cms-java-garbage-collector/ Jeremy D. Branham Tel: **DOTNET -Original Message- From: Branham, Jeremy [HR] Sent: Friday, May 30, 2014 11:52 AM To: solr-user@lucene.apache.org Subject: FW: suspect SOLR query from D029 (SOLR master) We saw the file descriptors peak out and full GCs running causing DOS on our SOLR servers this morning. * Does this stack trace give enough information for some ideas? * solr-spec 4.5.1-SNAPSHOT * solr-impl 4.5.1-SNAPSHOT ${svnversion} - kx101435 - 2013-11-04 17:39:36 * lucene-spec 4.5.1 * lucene-impl 4.5.1 1533280 - mark - 2013-10-17 21:40:03 We also saw some IO connection exceptions in the SOLR log - 2014-05-30 10:24:20,334 ERROR - SolrDispatchFilter - null:org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,334 ERROR - SolrCore - org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://myserver.com:8080/svc/solr/wdsp 2014-05-30 10:24:20,333 ERROR - SolrCore - org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,335 ERROR - SolrCore - org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,336 ERROR - SolrDispatchFilter - null:org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,335 ERROR - SolrDispatchFilter - null:org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking
FW: suspect SOLR query from D029 (SOLR master)
We saw the file descriptors peak out and full GCs running causing DOS on our SOLR servers this morning. * Does this stack trace give enough information for some ideas? * solr-spec 4.5.1-SNAPSHOT * solr-impl 4.5.1-SNAPSHOT ${svnversion} - kx101435 - 2013-11-04 17:39:36 * lucene-spec 4.5.1 * lucene-impl 4.5.1 1533280 - mark - 2013-10-17 21:40:03 We also saw some IO connection exceptions in the SOLR log - 2014-05-30 10:24:20,334 ERROR - SolrDispatchFilter - null:org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,334 ERROR - SolrCore - org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://myserver.com:8080/svc/solr/wdsp 2014-05-30 10:24:20,333 ERROR - SolrCore - org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,335 ERROR - SolrCore - org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,336 ERROR - SolrDispatchFilter - null:org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,335 ERROR - SolrDispatchFilter - null:org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsp Jeremy D. Branham Tel: **DOTNET From: Worley, Chris [HR] Sent: Friday, May 30, 2014 11:38 AM To: Stoner, Susan [HR]; Branham, Jeremy [HR] Cc: Barrett, Kevin B [HR]; Aldrich, Daniel J [HR]; Duncan, Horace W [HR] Subject: suspect SOLR query from D029 (SOLR master) Looks like a suspect SOLR query was executed at approx. 8:54am this morning on the SOLR master server (D029) that caused the java GC to go through the roof. See below: 2014-05-30 08:54:17,092 ERROR - SolrDispatchFilter - null:java.lang.OutOfMemoryError: GC overhead limit exceeded at org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum$Frame.loadBlock(BlockTreeTermsReader.java:2377) at org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum.seekExact(BlockTreeTermsReader.java:1698) at org.apache.lucene.index.TermContext.build(TermContext.java:95) at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:166) at org.apache.lucene.search.BooleanQuery$BooleanWeight.init(BooleanQuery.java:183) at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384) at org.apache.lucene.search.BooleanQuery$BooleanWeight.init(BooleanQuery.java:183) at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384) at org.apache.lucene.search.BooleanQuery$BooleanWeight.init(BooleanQuery.java:183) at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384) at org.apache.lucene.search.BooleanQuery$BooleanWeight.init(BooleanQuery.java:183) at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384) at org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82) at org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:690) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297) at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1501) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1367) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:474) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:434) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:703) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:406) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206
RE: suspect SOLR query from D029 (SOLR master)
We've switched to CMS GC to see if there is any improvement. Looking at this use case, G1GC might have been a better option, but we are running JDK 1.6 http://blog.sematext.com/2013/06/24/g1-cms-java-garbage-collector/ Jeremy D. Branham Tel: **DOTNET -Original Message- From: Branham, Jeremy [HR] Sent: Friday, May 30, 2014 11:52 AM To: solr-user@lucene.apache.org Subject: FW: suspect SOLR query from D029 (SOLR master) We saw the file descriptors peak out and full GCs running causing DOS on our SOLR servers this morning. * Does this stack trace give enough information for some ideas? * solr-spec 4.5.1-SNAPSHOT * solr-impl 4.5.1-SNAPSHOT ${svnversion} - kx101435 - 2013-11-04 17:39:36 * lucene-spec 4.5.1 * lucene-impl 4.5.1 1533280 - mark - 2013-10-17 21:40:03 We also saw some IO connection exceptions in the SOLR log - 2014-05-30 10:24:20,334 ERROR - SolrDispatchFilter - null:org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,334 ERROR - SolrCore - org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://myserver.com:8080/svc/solr/wdsp 2014-05-30 10:24:20,333 ERROR - SolrCore - org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,335 ERROR - SolrCore - org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,336 ERROR - SolrDispatchFilter - null:org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsc 2014-05-30 10:24:20,335 ERROR - SolrDispatchFilter - null:org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http:// myserver.com:8080/svc/solr/wdsp Jeremy D. Branham Tel: **DOTNET From: Worley, Chris [HR] Sent: Friday, May 30, 2014 11:38 AM To: Stoner, Susan [HR]; Branham, Jeremy [HR] Cc: Barrett, Kevin B [HR]; Aldrich, Daniel J [HR]; Duncan, Horace W [HR] Subject: suspect SOLR query from D029 (SOLR master) Looks like a suspect SOLR query was executed at approx. 8:54am this morning on the SOLR master server (D029) that caused the java GC to go through the roof. See below: 2014-05-30 08:54:17,092 ERROR - SolrDispatchFilter - null:java.lang.OutOfMemoryError: GC overhead limit exceeded at org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum$Frame.loadBlock(BlockTreeTermsReader.java:2377) at org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum.seekExact(BlockTreeTermsReader.java:1698) at org.apache.lucene.index.TermContext.build(TermContext.java:95) at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:166) at org.apache.lucene.search.BooleanQuery$BooleanWeight.init(BooleanQuery.java:183) at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384) at org.apache.lucene.search.BooleanQuery$BooleanWeight.init(BooleanQuery.java:183) at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384) at org.apache.lucene.search.BooleanQuery$BooleanWeight.init(BooleanQuery.java:183) at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384) at org.apache.lucene.search.BooleanQuery$BooleanWeight.init(BooleanQuery.java:183) at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384) at org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82) at org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:690) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297) at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1501) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1367) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:474) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:434) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859) at org.apache.solr.servlet.SolrDispatchFilter.execute