Re: big perf-difference between solr-server vs. SOlrJ req.process(solrserver)
On Dec 27, 2007 9:45 AM, Britske [EMAIL PROTECTED] wrote: I am using SolrJ to communicate with SOLR. My Solr-queries perform within range (between 50 ms and 300 ms) by looking at the solr log as ouputted on my (windows) commandline. However I discovered that the following command at all times takes significantly longer than the number outputted in the solr-log, (sometimes about 400% longer): It's probably due to stored field retrieval. The time in the response includes everything except the time to write the response (since it appears at the beginning). Writing the response involves reading the stored fields of documents (this was done to allow one to stream a large number of documents w/o having them all in memory). SolrJ's parsing of the response should be a relatively small constant cost. -Yonik
Re: An observation on the Too Many Files Open problem
Setting maxBufferedDocs to something smaller (say, 300), might be a better way of limiting your memory usage. I have difficulties with the odd huge document when using the default maxBufferedDocs=1000 (in the next Solr version, there should be an option to limit indexing based on memory usage and not #docs). There is also an option to flush pending deletes every X docs in trunk, which might make a difference if you are overwriting millions of docs. The autocommit problem seems odd to me: I definitely used it in 1.2 (maxDocs only). There have been a few bugs fixed in trunk, though (if that is an option). -Mike On 26-Dec-07, at 5:48 AM, Mark Baird wrote: Well when I wasn't sending regular commits I was getting out of memory exceptions from Solr fairly often, which I assume is due to the size of the documents I'm sending. I'd love to set the autocommit in solrconfig.xml and not worry about sending commits on the client side, but autocommit doesn't seem to work at all for me. I'm aware of the maxTime bug but I've tried setting maxTime to 0, and completely leaving it out of solrconfig.xml all together, but no matter what I try my Solr server never does an autocommit. Yes I'm just using one Solr instance for indexing and searching currently. However this is just in a development environment still. We just started looking at Solr about a month ago and haven't gone to production with anything yet. We will probably have separate instances for indexing and searching by the time we go live. Mark On Dec 24, 2007 4:20 PM, Otis Gospodnetic [EMAIL PROTECTED] wrote: Mark, Another question to ask is: do you *really* need to be calling commit every 300 docs? Unless you really need searchers to see your 300 new docs, you don't need to commit. Just optimize + commit at the end of your whole batch. Lowering the mergeFactor is the right thing to do. Out of curiosity, are you using a single instance of Solr for both indexing and searching? Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Mark Baird [EMAIL PROTECTED] To: Solr Mailing List solr-user@lucene.apache.org Sent: Monday, December 24, 2007 7:25:00 PM Subject: An observation on the Too Many Files Open problem Running our Solr server (latest 1.2 Solr release) on a Linux machine we ran into the Too Many Open Files issue quite a bit. We've since changed the ulimit max filehandle setting, as well as the Solr mergeFactor setting and haven't been running into the problem anymore. However we are seeing some behavior from Solr that seems a little odd to me. When we are in the middle of our batch index process and we run the lsof command we see a lot of open file handles hanging around that reference Solr index files that have been deleted by Solr and no longer exist on the system. The documents we are indexing are potentially very large, so due to various memory constraints we only send 300 docs to Solr at a time. With a commit between each set of 300 documents. Now one of the things that I read may cause old file handles to hang around was if you had an old IndexReader still open pointing to those old files. However whenever you issue a commit to the server it is supposed to close the old IndexReader and open a new one. So my question is, when the Reader is being closed due to a commit, what exactly is happening? Is it just being set to null and a new instance being created? I'm thinking the reader may be sitting around in memory for a while before the garbage collector finally gets to it, and in that time it is still holding those files open. Perhaps an explicit method call that closes any open file handles should occur before setting the reference to null? After looking at the code, it looks like reader.close() is explicitly being called as long as the closeReader property in SolrIndexSearcher is set to true, but I'm not sure how to check if that is always getting set to true or not. There is one constructor of SolrIndexSearcher that sets it to false. Any insight here would be appreciated. Are stale file handles something I should just expect from the JVM? I've never ran into the Too Many Files Open exception before, so this is my first time looking at the lsof command. Perhaps I'm reading too much into the data it's showing me. Mark Baird
RE: big perf-difference between solr-server vs. SOlrJ req.process(solrserver)
As a reference: I have several million records where are about 20 fields. One of them is 100-1k bytes, and the rest are 20-50 bytes. There is a reliable 5% performance difference between pulling just the unique key field and pulling all of the fields. -Original Message- From: Geert-Jan Brits [mailto:[EMAIL PROTECTED] Sent: Thursday, December 27, 2007 8:44 AM To: solr-user@lucene.apache.org Subject: Re: big perf-difference between solr-server vs. SOlrJ req.process(solrserver) yeah, that makes sense. so, in in all, could scanning all the fields and loading the 10 fields add up to cost about the same or even more as performing the intial query? (Just making sure) I am wondering if the following change to the schema would help in this case: current setup: It's possible to have up to 2000 product-variants. each product-variant has: - 1 price field (stored / indexed) - 1 multivalued field which contains product-variant characteristics (strored / not indexed). This adds up to the 4000 fields described. Moreover there are some fields on the product level but these would contibute just a tiny bit to the overall scanning / loading costs (about 50 -stored and indexed- fields in total) possible new setup (only the changes) : - index but not store the price-field. - store the price as just another one of the product-variant characteristics in the multivalued product-variant field. as a result this would bring back the maximum number of stored fields to about 2050 from 4050 and thereby about halving scanning / loading costs while leaving the current quering-costs intact. Indexing costs would increase a bit. Would you expect the same performance gain? Thanks, Geert-Jan 2007/12/27, Yonik Seeley [EMAIL PROTECTED]: On Dec 27, 2007 11:01 AM, Britske [EMAIL PROTECTED] wrote: after inspecting solrconfig.xml I see that I already have enabled lazy field loading by: enableLazyFieldLoadingtrue/enableLazyFieldLoading (I guess it was enabled by default) Since any query returns about 10 fields (which differ from query to query) , would this mean that only these 10 of about 2000-4000 fields are retrieved / loaded? Yes, but that's not the whole story. Lucene stores all of the fields back-to-back with no index (there is no random access to particular stored fields)... so all of the fields must be at least scanned. -Yonik
Cannot start SVN head Solr 1.3
Hello! I'm having a horrible time getting the current svn head build of Solr to run. I even remembered to do an 'ant clean' this time.. but no luck. I have set up solr_home via a JAVA_OPTS flag, and am using Tomcat 6... [EMAIL PROTECTED]:/opt/tomcat]$ echo $JAVA_OPTS -Dsolr.solr.home=/opt/solr Here is the entire deployment log from Tomcat. I also picked out the errors (right here).. SEVERE: org.apache.solr.common.SolrException: Error loading class 'solr.XppUpdateRequestHandler' at org .apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java: 206) at org .apache .solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:211) at org .apache .solr .util.plugin.AbstractPluginLoader.create(AbstractPluginLoader.java:83) at org.apache.solr.core.RequestHandlers$1.create(RequestHandlers.java: 152) at org.apache.solr.core.RequestHandlers$1.create(RequestHandlers.java: 137) at org .apache .solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java: 140) at org .apache .solr.core.RequestHandlers.initHandlersFromConfig(RequestHandlers.java: 169) at org.apache.solr.core.SolrCore.init(SolrCore.java:329) at org .apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java: 85)... SEVERE: Could not start SOLR. Check solr/home property org.apache.solr.common.SolrException: Unknown Search Component: org.apache.solr.handler.component.QueryComponent at org.apache.solr.core.SolrCore.getSearchComponent(SolrCore.java:507) at org.apache.solr.handler.SearchHandler.inform(SearchHandler.java:115) at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java: 243) at org.apache.solr.core.SolrCore.init(SolrCore.java:350) at org .apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:85) Solr home, /opt/solr, is, for sure, good - unless something has changed in the last few SVN commits. I didn't see any changes. Any help would be most appreciated. --- INFO: Deploying web application archive solr.war Dec 27, 2007 4:00:01 PM org.apache.solr.servlet.SolrDispatchFilter init INFO: SolrDispatchFilter.init() Dec 27, 2007 4:00:01 PM org.apache.solr.core.SolrResourceLoader locateInstanceDir INFO: Using JNDI solr.home: /opt/solr Dec 27, 2007 4:00:01 PM org.apache.solr.servlet.SolrDispatchFilter init INFO: looking for multicore.xml: /opt/solr/multicore.xml Dec 27, 2007 4:00:01 PM org.apache.solr.core.SolrResourceLoader locateInstanceDir INFO: Using JNDI solr.home: /opt/solr Dec 27, 2007 4:00:01 PM org.apache.solr.core.SolrResourceLoader init INFO: Solr home set to '/opt/solr/' Dec 27, 2007 4:00:01 PM org.apache.solr.core.SolrResourceLoader createClassLoader INFO: Reusing parent classloader Dec 27, 2007 4:00:01 PM org.apache.solr.core.SolrConfig init INFO: Loaded SolrConfig: solrconfig.xml Dec 27, 2007 4:00:01 PM org.apache.solr.core.SolrCore init INFO: [core] Opening new SolrCore at /opt/solr/, dataDir=/opt/solr/data Dec 27, 2007 4:00:01 PM org.apache.solr.schema.IndexSchema readSchema INFO: Reading Solr Schema Dec 27, 2007 4:00:01 PM org.apache.solr.core.SolrResourceLoader locateInstanceDir INFO: Using JNDI solr.home: /opt/solr Dec 27, 2007 4:00:01 PM org.apache.solr.core.SolrResourceLoader init INFO: Solr home set to '/opt/solr/' Dec 27, 2007 4:00:01 PM org.apache.solr.core.SolrResourceLoader createClassLoader INFO: Reusing parent classloader Dec 27, 2007 4:00:01 PM org.apache.solr.schema.IndexSchema readSchema INFO: Schema name=Zappos Beta Dec 27, 2007 4:00:01 PM org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created string: org.apache.solr.schema.StrField Dec 27, 2007 4:00:01 PM org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created boolean: org.apache.solr.schema.BoolField Dec 27, 2007 4:00:01 PM org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created integer: org.apache.solr.schema.IntField Dec 27, 2007 4:00:01 PM org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created long: org.apache.solr.schema.LongField Dec 27, 2007 4:00:01 PM org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created float: org.apache.solr.schema.FloatField Dec 27, 2007 4:00:01 PM org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created double: org.apache.solr.schema.DoubleField Dec 27, 2007 4:00:01 PM org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created sint: org.apache.solr.schema.SortableIntField Dec 27, 2007 4:00:01 PM org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created slong: org.apache.solr.schema.SortableLongField Dec 27, 2007 4:00:01 PM org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created sfloat: org.apache.solr.schema.SortableFloatField Dec 27, 2007 4:00:01 PM org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created sdouble: org.apache.solr.schema.SortableDoubleField Dec 27, 2007 4:00:01 PM
Re: Cannot start SVN head Solr 1.3
The XppUpdateRequestHandler was removed this afternoon... make sure your sorlconfig.xml does not include: requestHandler name=/update/xpp class=solr.XppUpdateRequestHandler / holler if you have problems! ryan Matthew Runo wrote: Hello! I'm having a horrible time getting the current svn head build of Solr to run. I even remembered to do an 'ant clean' this time.. but no luck. I have set up solr_home via a JAVA_OPTS flag, and am using Tomcat 6... [EMAIL PROTECTED]:/opt/tomcat]$ echo $JAVA_OPTS -Dsolr.solr.home=/opt/solr Here is the entire deployment log from Tomcat. I also picked out the errors (right here).. SEVERE: org.apache.solr.common.SolrException: Error loading class 'solr.XppUpdateRequestHandler' at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:206) at org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:211) at org.apache.solr.util.plugin.AbstractPluginLoader.create(AbstractPluginLoader.java:83) at org.apache.solr.core.RequestHandlers$1.create(RequestHandlers.java:152) at org.apache.solr.core.RequestHandlers$1.create(RequestHandlers.java:137) at org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:140) at org.apache.solr.core.RequestHandlers.initHandlersFromConfig(RequestHandlers.java:169) at org.apache.solr.core.SolrCore.init(SolrCore.java:329) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:85)... SEVERE: Could not start SOLR. Check solr/home property org.apache.solr.common.SolrException: Unknown Search Component: org.apache.solr.handler.component.QueryComponent at org.apache.solr.core.SolrCore.getSearchComponent(SolrCore.java:507) at org.apache.solr.handler.SearchHandler.inform(SearchHandler.java:115) at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:243) at org.apache.solr.core.SolrCore.init(SolrCore.java:350) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:85) Solr home, /opt/solr, is, for sure, good - unless something has changed in the last few SVN commits. I didn't see any changes. Any help would be most appreciated. --- INFO: Deploying web application archive solr.war Dec 27, 2007 4:00:01 PM org.apache.solr.servlet.SolrDispatchFilter init INFO: SolrDispatchFilter.init() Dec 27, 2007 4:00:01 PM org.apache.solr.core.SolrResourceLoader locateInstanceDir INFO: Using JNDI solr.home: /opt/solr Dec 27, 2007 4:00:01 PM org.apache.solr.servlet.SolrDispatchFilter init INFO: looking for multicore.xml: /opt/solr/multicore.xml Dec 27, 2007 4:00:01 PM org.apache.solr.core.SolrResourceLoader locateInstanceDir INFO: Using JNDI solr.home: /opt/solr Dec 27, 2007 4:00:01 PM org.apache.solr.core.SolrResourceLoader init INFO: Solr home set to '/opt/solr/' Dec 27, 2007 4:00:01 PM org.apache.solr.core.SolrResourceLoader createClassLoader INFO: Reusing parent classloader Dec 27, 2007 4:00:01 PM org.apache.solr.core.SolrConfig init INFO: Loaded SolrConfig: solrconfig.xml Dec 27, 2007 4:00:01 PM org.apache.solr.core.SolrCore init INFO: [core] Opening new SolrCore at /opt/solr/, dataDir=/opt/solr/data Dec 27, 2007 4:00:01 PM org.apache.solr.schema.IndexSchema readSchema INFO: Reading Solr Schema Dec 27, 2007 4:00:01 PM org.apache.solr.core.SolrResourceLoader locateInstanceDir INFO: Using JNDI solr.home: /opt/solr Dec 27, 2007 4:00:01 PM org.apache.solr.core.SolrResourceLoader init INFO: Solr home set to '/opt/solr/' Dec 27, 2007 4:00:01 PM org.apache.solr.core.SolrResourceLoader createClassLoader INFO: Reusing parent classloader Dec 27, 2007 4:00:01 PM org.apache.solr.schema.IndexSchema readSchema INFO: Schema name=Zappos Beta Dec 27, 2007 4:00:01 PM org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created string: org.apache.solr.schema.StrField Dec 27, 2007 4:00:01 PM org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created boolean: org.apache.solr.schema.BoolField Dec 27, 2007 4:00:01 PM org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created integer: org.apache.solr.schema.IntField Dec 27, 2007 4:00:01 PM org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created long: org.apache.solr.schema.LongField Dec 27, 2007 4:00:01 PM org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created float: org.apache.solr.schema.FloatField Dec 27, 2007 4:00:01 PM org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created double: org.apache.solr.schema.DoubleField Dec 27, 2007 4:00:01 PM org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created sint: org.apache.solr.schema.SortableIntField Dec 27, 2007 4:00:01 PM org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created slong: org.apache.solr.schema.SortableLongField Dec 27, 2007 4:00:01 PM org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created sfloat:
RAMDirectory
Is there a way to use RAMDirectory with SOLR?If you can point me to documentation that would be great. Thanks, S
Search query have AND operator with facet search not working
Hello everyone, I have implemented facet search with search in all fields, its working fine with the following url. http://localhost:8983/solr/select?q=cdfacet=truefacet.field=mediafacet.limit=-1start=0rows=15facet.mincount=1qt=LM Query : cd But when I tried with query which have AND, its not shows any results, my url as below. Is there are any issue with URL or any other, http://localhost:8983/solr/select?q=cd AND media:moviesfacet=truefacet.field=mediafacet.limit=-1start=0rows=15facet.mincount=1qt=LM Query : cd AND media:movies Please let me know, what should I do ? -- Thanks, Laxmilal menaria http://www.chambal.com/ http://www.minalyzer.com/ http://www.bucketexplorer.com/
