RE: Any option to NOT return stack trace in Solr response?
Hi Alex, Thanks for confirming my finding. When it comes to Solr interfacing to a client, I agree completely. However, I was hoping to limit the noise at Solr and not have to add extra code to filter out the exceptions. Just wondering, wouldn't it be a cleaner RESTFUL interface if instead of reporting the stack trace in response, Solr would return an error code and a basic message pointing back to Solr log for details such as stack trace. I am curious, what use case would it serve where one would require the stack trace in response? If there is interest, I could open an JIRA and come up with a patch. Regards, Koorosh -Original Message- From: Alexandre Rafalovitch [mailto:arafa...@gmail.com] Sent: Thursday, July 21, 2016 6:54 PM To: solr-user Subject: Re: Any option to NOT return stack trace in Solr response? I don't think there is a flag. But the bigger question is whether you are exposing Solr directly to the client? You should not be. You should have a middleware client that talks to Solr and then generates web UI or whatever. If you give untrusted access to Solr, there are too many things that can be done, starting from deleting the whole index. It might be possible to have a smart proxy and expose Solr with heavily filtered valid URLs, then you would need to scrub response. That's all I can think of without hacking and reregistering with your own response handler (probably not that hard). Regards, Alex. Newsletter and resources for Solr beginners and intermediates: http://www.solr-start.com/ On 22 July 2016 at 03:35, Koorosh Vakhshoori wrote: > Hi all, > Got a Solr 5.2.1 installation. I am getting following error response when > calling the TERMS component. Now the error is not the point, I know what is > going on in this instance. However, to address security concerns, I am trying > to have Solr truncate the stack trace in the response. Of course I would > still want Solr to log the error in its log file. What I was wondering, if > there is a flag or option I can set in solrconfig.xml globally or under TERMS > to omit the trace or just return ' java.lang.NullPointerException'? I have > looked at the source code and don't see anything relevant. However, I may > have missed something. Appreciated any suggestion and pointers. > > > > 500 > 5 > > > > java.lang.NullPointerException at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(Sear > chHandler.java:322) at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandle > rBase.java:143) at > org.apache.solr.core.SolrCore.execute(SolrCore.java:2067) at > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654) at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450) at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter > .java:227) at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter > .java:196) at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli > cationFilterChain.java:239) at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi > lterChain.java:206) at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli > cationFilterChain.java:239) at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi > lterChain.java:206) at > org.apache.catalina.filters.CorsFilter.handleNonCORS(CorsFilter.java:4 > 39) at > org.apache.catalina.filters.CorsFilter.doFilter(CorsFilter.java:178) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli > cationFilterChain.java:239) at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi > lterChain.java:206) at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa > lve.java:219) at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:136) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) > at > org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:526) > at > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078) > at > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:655) > at > org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:222) > at > org.apache.tomcat.util.net.NioEndpoint
Any option to NOT return stack trace in Solr response?
Hi all, Got a Solr 5.2.1 installation. I am getting following error response when calling the TERMS component. Now the error is not the point, I know what is going on in this instance. However, to address security concerns, I am trying to have Solr truncate the stack trace in the response. Of course I would still want Solr to log the error in its log file. What I was wondering, if there is a flag or option I can set in solrconfig.xml globally or under TERMS to omit the trace or just return ' java.lang.NullPointerException'? I have looked at the source code and don't see anything relevant. However, I may have missed something. Appreciated any suggestion and pointers. 500 5 java.lang.NullPointerException at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:322) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2067) at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.filters.CorsFilter.handleNonCORS(CorsFilter.java:439) at org.apache.catalina.filters.CorsFilter.doFilter(CorsFilter.java:178) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:136) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:526) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:655) at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:222) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1566) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1523) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) 500 Regards, Koorosh
RE: How for distributed search only log collective search response
It turns out there is a better way to do this. It does not require code change in Solr, if you are using log4j. However, you need to migrate to log4j.xml file format. The solution is to use the filter feature. Here is what my console appender looks like with the filter: Regards, Koorosh
How for distributed search only log collective search response
In my use case, I have a number of shards where a query would run as distributed search. I am not using Solr Cloud, I have just a Solr server. Now, when the search runs, I see one entry for each shard query as well as the finally collective search query response. As the results, I am ending up with a very noisy log. I don't care about individual shard queries, just the aggregate result. Is there a way to configure Solr so it would only log the final collective response? I believe this use case also applies to Solr Cloud. Looking at the Solr code, class SolrCore, I see the following lines is performing the logging: if (rsp.getToLog().size() > 0) { if (requestLog.isInfoEnabled()) { requestLog.info(rsp.getToLogAsString(logid)); } I was thinking of adding a flag that filter the distributed logs by looking at the 'params' and check for 'isShard=true' and if present don't log it. Any suggestion or comment? Is this something people would be interested in? Regards, Koorosh
Custom JUnit tests based on SolrTestCaseJ4 fails intermittently.
Hi, I am trying out the Solr Alpha release against some custom and Junit codes I have written. I am seeing my custom JUnit tests failing once in a while. The tests are based on Solr Junit test code where they are extending SolrTestCaseJ4. My guess is the Randomized Testing coming across some issue here. However not sure what the source of the problem is. I noticed the value of 'codec' is null for failed cases, but I am setting the luceneMatchVersion value in solrconfig.xml as bellow: ${tests.luceneMatchVersion:LUCENE_CURRENT} I am including the test outputs for both scenarios here. Any help or pointer appreciated. Thanks, Koorosh Here is the output of Junit test which failes when running it from Eclipse: NOTE: test params are: codec=null, sim=null, locale=null, timezone=(null) NOTE: Windows 7 6.1 amd64/Sun Microsystems Inc. 1.6.0_21 (64-bit)/cpus=4,threads=1,free=59414480,total=63242240 NOTE: All tests run in this JVM: [TestDocsHandler] Jul 18, 2012 3:55:25 PM com.carrotsearch.randomizedtesting.RandomizedRunner runSuite SEVERE: Panic: RunListener hook shouldn't throw exceptions. java.lang.NullPointerException at org.apache.lucene.util.RunListenerPrintReproduceInfo.reportAdditionalFailureInfo(RunListenerPrintReproduceInfo.java:159) at org.apache.lucene.util.RunListenerPrintReproduceInfo.testRunFinished(RunListenerPrintReproduceInfo.java:104) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:634) at com.carrotsearch.randomizedtesting.RandomizedRunner.access$400(RandomizedRunner.java:132) at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:551) Here is the output for the same test where it is successful: 24 T11 oas.SolrTestCaseJ4.initCore initCore Creating dataDir: C:\Users\xuser\AppData\Local\Temp\solrtest-TestDocsHandler-1342651924084 43 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx) 43 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: solr-gold/solr-extraction 45 T11 oasc.SolrResourceLoader. new SolrResourceLoader for deduced Solr Home: 'solr-gold/solr-extraction\' 284 T11 oasc.SolrConfig. Using Lucene MatchVersion: LUCENE_40 429 T11 oasc.SolrConfig. Loaded SolrConfig: solrconfig-dow.xml 434 T11 oass.IndexSchema.readSchema Reading Solr Schema 443 T11 oass.IndexSchema.readSchema Schema name=SolvNet Common core 522 T11 oass.IndexSchema.readSchema default search field in schema is indexed_content 524 T11 oass.IndexSchema.readSchema query parser default operator is AND 525 T11 oass.IndexSchema.readSchema unique key field: id 616 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx) 617 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: solr-gold/solr-extraction 617 T11 oasc.SolrResourceLoader. new SolrResourceLoader for directory: 'solr-gold/solr-extraction\' 618 T11 oasc.CoreContainer. New CoreContainer 994682772 642 T11 oasc.SolrCore. [collection1] Opening new SolrCore at solr-gold/solr-extraction\, dataDir=C:\Users\koo\AppData\Local\Temp\solrtest-TestDocsHandler-1342651924084\ 642 T11 oasc.SolrCore. JMX monitoring not detected for core: collection1 648 T11 oasc.SolrCore.getNewIndexDir WARNING New index directory detected: old=null new=C:\Users\koo\AppData\Local\Temp\solrtest-TestDocsHandler-1342651924084\index/ 648 T11 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory 'C:\Users\koo\AppData\Local\Temp\solrtest-TestDocsHandler-1342651924084\index' doesn't exist. Creating new index... 742 T11 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1 commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@44023756 lockFactory=org.apache.lucene.store.NativeFSLockFactory@21ed5459),segFN=segments_1,generation=1,filenames=[segments_1] 743 T11 oasc.SolrDeletionPolicy.updateCommits newest commit = 1 871 T11 oasc.RequestHandlers.initHandlersFromConfig created /update/javabin: solr.BinaryUpdateRequestHandler 875 T11 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler 878 T11 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.XmlUpdateRequestHandler 878 T11 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers 886 T11 oasc.RequestHandlers.initHandlersFromConfig created /update/extract: com.synopsys.ies.solr.backend.handler.extraction.SolvNetExtractingRequestHandler 891 T11 oasc.RequestHandlers.initHandlersFromConfig WARNING Multiple requestHandler registered to the same name: standard ignoring: org.apache.solr.handler.StandardRequestHandler 892 T11 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.SearchHandler 892 T11 oasc.RequestHandlers.initHandlersFromConfig created employee: solr.SearchHandler 892 T11 oasc.RequestHandlers.initHandlersFromConf
Solr 4.0 ALPHA: AbstractSolrTestCase depending on LuceneTestCase
Hi, I have been developing extensions to SOLR code using 4.0 truck. For JUnit testing I am extending AbstractSolrTestCase which in the ALPHA release is located in JAR apache-solr-test-framework-4.0.0-ALPHA.jar. However, this class extends LuceneTestCase which comes from JAR lucene-test-framework-4.0-SNAPSHOT.jar. In the ALPHA release the later JAR is not shipped or I can't find it. My question is which class should I use for testing customized/extensions to SOLR/LUCENE code? Is there a better way of doing this without build the lucene-test-framework-4.0-SNAPSHOT.jar from the source code? Thanks, Koorosh -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-0-ALPHA-AbstractSolrTestCase-depending-on-LuceneTestCase-tp3995639.html Sent from the Solr - User mailing list archive at Nabble.com.
ContentStreamUpdateRequest method addFile in 4.0 release.
In latest 4.0 release, the addFile() method has a new argument 'contentType': addFile(File file, String contentType) In context of Solr Cell how should addFile() method be called? Specifically I refer to the Wiki example: ContentStreamUpdateRequest up = new ContentStreamUpdateRequest("/update/extract"); up.addFile(new File("mailing_lists.pdf")); up.setParam("literal.id", "mailing_lists.pdf"); up.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true); result = server.request(up); assertNotNull("Couldn't upload mailing_lists.pdf", result); rsp = server.query( new SolrQuery( "*:*") ); Assert.assertEquals( 1, rsp.getResults().getNumFound() ); given at URL: http://wiki.apache.org/solr/ExtractingRequestHandler Since Solr Cell is calling Tika under the hood, doesn't the file content-type is already identified by Tika? Looking at the code, it seems passing NULL would do the job, is that correct? Also for Solr Cell, is the ContentStreamUpdateRequest class is the right one to use or there is a different class that is more appropriate here? Thanks -- View this message in context: http://lucene.472066.n3.nabble.com/ContentStreamUpdateRequest-method-addFile-in-4-0-release-tp3988344.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Display of highlighted search result should start with the beginning of the sentence that contains the search string.
Hi Koji, I am Shyam's coworker. After some looking into this issue, I believe the problem of chopped word has to do with org.apache.lucene.search.vectorhighlight.SimpleFragListBuilder class' 'margin' field. It is set to 6 by default. My understanding is having margin value of greater than zero results in truncated word when the highlighted term is too close to beginning of a document. I was able to reset the 'margin' field by creating my custom version of org.apache.solr.highlight.SimpleFragListBuilder and passing zero for 'margin' when calling the Lucene's SimpleFragListBuilder constructor. My testing shows the problem has been fixed. Do you concur? Now couple of questions. Not sure what the purpose of this field is, could you give the use case for it? Also could it be exposed as a parameter in Solr so it could be set to some other value? Thanks, Koorosh -- View this message in context: http://lucene.472066.n3.nabble.com/Display-of-highlighted-search-result-should-start-with-the-beginning-of-the-sentence-that-contains-t-tp3722912p3820516.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Indexing leave behind write.lock file.
Here is how I got SolrJ to delete the write.lock file. I switched to the CoreContainer's remove() method. So the new code is: ... SolrCore curCore = container.remove("core1"); curCore.close(); Now my understanding for why it is working. Based on Solr source code, the issue had to do with the core's reference count not ending up at zero when the close() method is called. The getCore() method increments the reference count while remove() doesn't. Since the close() method decrements the count first and if and only if the count is zero it would unlock the core, i.e. remove the write.lock. Regards, Koorosh -- View this message in context: http://lucene.472066.n3.nabble.com/Indexing-leave-behind-write-lock-file-tp3701915p3705554.html Sent from the Solr - User mailing list archive at Nabble.com.
Indexing leave behind write.lock file.
Hi, I am using SolrJ to reindex a core in a multiCore setup. The general flow of my program is as follows (pseudo code): String solrHome = "/opt/solr/home"; File solrXml = new File( solrHome, "solr.xml" ); container = new CoreContainer(); container.load(solrHome, solrXml); SolrServer solr = new EmbeddedSolrServer(container, "core1"); solr.deleteByQuery("*:*"); SolrInputDocument doc1 = new SolrInputDocument(); doc1.addField( "id", "id1", 1.0f ); doc1.addField( "name", "doc1", 1.0f ); Collection docs = new ArrayList(); docs.add( doc1 ); solr.commit(); SolrCore curCore = container.getCore("core1"); curCore.close(); I thought for sure by calling close(), I would also be releasing all associated resources including the lock on the core that is I would getting rid of the write.lock file. I am using Solr 4.0 code from the development truck which is about a month old. Any suggestion here appreciated. Regards, Koorosh
Question on XPATH use in Solr Cell.
I am new to both Solr and Cell, so sorry if I am misusing some of the terminologies. So the problem I am trying to solve is to index a PDF document using Solr Cell where I want to exclude part of it via XPATH. I am using Solr release 3.1. When researching the user list, I came across one entry on this topic titled 'XPath query support in Solr Cell' which clarify one issue, but still I am having problem getting what I want. Here is what I have done so far: First, I started by executing the following 'CURL' command to see what I would get: curl "http://localhost:8983/solr/docs/update/extract?literal.id=123&xpath=/xhtml:html/xhtml:body/xhtml:div/descendant:node()&extractOnly=true" -F "file=@/docs/test.pdf" This worked fine. Next I tried getting the first DIV element by modifying the XPATH query as follows: curl "http://localhost:8983/solr/docs/update/extract?literal.id=123&xpath=/xhtml:html/xhtml:body/xhtml:div\[1\]/descendant:node()&extractOnly=true" -F "file=@/docs/test.pdf" Note, I am escaping the '[]', I even tried using their encoded values %5B and %5D. It ran, but it did not match anything. Here is was I got: 0627 fileVersion A-2007. 122009-08-12T17:07:27ZTest title.FrameMaker 7.1187 2009-08-12T17:07:27ZTest Documentapplication/octet-streamWed Aug 12 10:07:2 7 PDT 20091372769test.pdfAcrobat Di stiller 7.0.5 (Windows)2007application/pdfTest On a different track I explored what could be an XPATH expression for my purpose. Here I have something that should get me there most of the way: //xhtml:body/xhtml:div\[not(contains(p,'EXCLUDE TEXT'))\] I independently validated the XPATH expression at following URL: http://www.whitebeam.org/library/guide/TechNotes/xpathtestbed.rhtm As was suggested in previously mentioned posting. Any suggestion and help is greatly appreciated.