Re: fieldCache problem OOM exception
The current status of my installation is that with some tweeking of JAVA I get a runtime of about 2 weeks until OldGen (14GB) is filled to 100 percent and won't free anything even with FullGC. The part of fieldCache in a HeapDump to that time is over 80 percent from the whole heap (20GB). And that is what eats up all OldGen until OOM. Next week I will start with tomcat 6.x to see how that one behaves, but there isn't any hope. It is just a different container which wouldn't change anything about how Lucene eats up memory with fieldCache. After digging through all the code, logging and debugging I can say that it seams to be not a memory leak. Solr is using the fieldCache from Lucene under the hood of the servlet container. The fieldCache grows until everything cachable is in memory or OOM is reached, what ever comes first. The description says: Provides introspection of the Lucene FieldCache, this is **NOT** a cache that is managed by Solr. So it seams to be a Lucene problem. As a matter of fact and due to this limitation solr can't be used with a single huge index. I don't know how other applications which are using Lucene and it's fieldCache (and there are a lot of them) are handling this and how they take care of the size of the fieldCache. And, I currently don't know how to calculate the limit. Say for example: the size of *.tii and *.tis file in the index should be the -Xmx size of your JAVA to be save with fieldCache and OOM. May be an expert can give more detailed info about fieldCache and its possible maximum size. Some data about our index: -rw-r--r-- 1 solr users 84448291214 19. Jul 10:43 _12jl.fdt -rw-r--r-- 1 solr users 236458468 19. Jul 10:43 _12jl.fdx -rw-r--r-- 1 solr users1208 19. Jul 10:30 _12jl.fnm -rw-r--r-- 1 solr users 19950615826 19. Jul 11:20 _12jl.frq -rw-r--r-- 1 solr users 532031548 19. Jul 11:20 _12jl.nrm -rw-r--r-- 1 solr users 20616887682 19. Jul 11:20 _12jl.prx -rw-r--r-- 1 solr users 291149087 19. Jul 11:20 _12jl.tii -rw-r--r-- 1 solr users 30850743727 19. Jul 11:20 _12jl.tis -rw-r--r-- 1 solr users 20 9. Jun 11:11 segments.gen -rw-r--r-- 1 solr users 274 19. Jul 11:20 segments_pl Size: 146,15 GB Docs: 29.557.308 Regards, Bernd Am 22.07.2011 00:10, schrieb Santiago Bazerque: Hello Erick, I have a 1.7MM documents, 3.6GB index. I also hava an unusual amount of dynamic fields, that I use for sorting. My FieldCache currently has about 13.000 entries, even though my index only has 1-3 queries per second. Each query sorts by two dynamic fields, and facets on 3-4 fields that are fixed. These latter fields are always in the field cache, what I find suspicious is the other ~13.000 that are sitting there. I am using a 32GB heap, and I am seeing periodical OOM errors (I didn't spot a regular pattern as Bernd did, but haven't increased RAM as methodically as he has). If you need any more info, I'll be glad to post it to the list. Best, Santiago On Fri, Jun 17, 2011 at 9:13 AM, Erick Ericksonerickerick...@gmail.comwrote: Sorry, it was late last night when I typed that... Basically, if you sort and facet on #all# the fields you mentioned, it should populate the cache in one go. If the problem is that you just have too many unique terms for all those operations, then it should go bOOM. But, frankly, that's unlikely, I'm just suggesting that to be sure the easy case isn't the problem. Take a memory snapshot at that point just to see, it should be a high-water mark. The fact that you increase the heap and can then run for longer is extremely suspicious, and really smells like a memory issue, so we'd like to pursue it. I'd be really interested if anyone else is seeing anything similar, these are the scary ones... Best Erick On Fri, Jun 17, 2011 at 3:09 AM, Bernd Fehling bernd.fehl...@uni-bielefeld.de wrote: Hi Erik, I will take some memory snapshots during the next week, but how can it be to get OOMs with one query? - I started with 6g for JVM -- 1 day until OOM. - increased to 8 g -- 2 days until OOM - increased to 10g -- 3.5 days until OOM - increased to 16g -- 5 days until OOM - currently 20g -- about 7 days until OOM Starting the system takes about 3.5g and goes up to about 4g after a while. The only dirty workaround so far is to restart the whole system after 5 days. Not really nice. The problem seams to be fieldCache which is under the hood of jetty. Do you know of any sizing features for fieldCache to limit the memory consumption? Regards Bernd Am 17.06.2011 03:37, schrieb Erick Erickson: Well, if my theory is right, you should be able to generate OOMs at will by sorting and faceting on all your fields in one query. But Lucene's cache should be garbage collected, can you take some memory snapshots during the week? It should hit a point and stay steady there. How much memory are you giving your JVM? It looks like a lot given your memory snapshot. Best Erick On Thu, Jun 16, 2011 at 3:01 AM,
RE: What is the different?
Hi, Have you check the queries by using the debugQuery=true parameter ? This could give some hints of what is searched in both cases. Pierre -Message d'origine- De : cnyee [mailto:yeec...@gmail.com] Envoyé : vendredi 22 juillet 2011 05:14 À : solr-user@lucene.apache.org Objet : What is the different? Hi, I have two queries: (1) q = (change management) (2) q = (change management) AND domain_ids:(0^1.3 OR 1) The purpose of the (2) is to boost the records with domain_ids=0. In my database all records has domain_ids = 0 or 1, so domains_ids:(0 or 1) will always returns the full database. Now my questions is - query (2) returns 5000+ results, but query (1) returns 700+ results. Can somebody enlighten me on what is the reasons behind such a vast different in number of results? Many thanks in advance. Yee -- View this message in context: http://lucene.472066.n3.nabble.com/What-is-the-different-tp3190278p3190278.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Geospatial queries in Solr
Jamie, You are using the field named point which is based on PointFieldType. Keep in mind that just because this field type is named this way, does *not* mean at all that other fields don't hold points, or that this one is especially suited to it. Arguably this one is named poorly. This field type is loosely based on Solr's LatLonType but it doesn't yet fully implement capabilities like crossing the dateline and shapes other than a bounding box. I've enhanced the code to log a warning if the query shape is expanded to the bounding box. Any way, it's a currently a 2nd class citizen in LSP. I recommend you use the field named geohash which uses a field type name starting with Recursive Note that the same query you did with this field returned the polygon of Russia. There is an outstanding box in which we don't properly handle polygons crossing the dateline; other shapes are fine. ~ David - Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book -- View this message in context: http://lucene.472066.n3.nabble.com/Geospatial-queries-in-Solr-tp3183374p3190762.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: embeded solrj doesn't refresh index
Are u indexing with full import? In case yes and the resultant index has similar num of docs (that the one you had before) try setting reopenReaders to false in solrconfig.xml * You have to send the comit, of course. -- View this message in context: http://lucene.472066.n3.nabble.com/embeded-solrj-doesn-t-refresh-index-tp3184321p3190892.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Culr Tika not working with blanks into literal.field
Hi. Is it possible setting fields with blaks values using extract update ? Thanks From: pacopera...@hotmail.com To: solr-user@lucene.apache.org Subject: Culr Tika not working with blanks into literal.field Date: Wed, 20 Jul 2011 12:53:18 + Hi. I'm trying to index binary documents with curl and Tika for extracting text. The problem is that when I set the value of a field with spaces blanks using the input parameter: literal.fieldname=value, the document is not indexed. The sentence I send is the follow: curl http://localhost:8983/solr/update/extract?literal.id=doc1\literal.url=/mnt/windows/Ofertas/2006 Portal Intranet/DOCUMENTACION/datos.doc\uprefix=attr_\fmap.content=text\commit=true -F myfile=\@/mnt/windows/Ofertas/DOCUMENTACION/datos.doc That is literal.url=value with blanks apparently is not working
RE: Solr 3.3: Exception in thread Lucene Merge Thread #1
mdz-munich wrote: Yeah, indeed. But since the VM is equipped with plenty of RAM (22GB) and it works so far (Solr 3.2) very well with this setup, I AM slightly confused, am I? Maybe we should LOWER the dedicated Physical Memory? The remaining 10GB are used for a second tomcat (8GB) and the OS (Suse). As far as I understand NIO (mostly un-far), this package can directly use the most efficient operations of the underlying platform. After three days of banging my head against the wall: Problem solved! It seems there was not enough Memory left for NIO. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-3-3-Exception-in-thread-Lucene-Merge-Thread-1-tp3185248p3190916.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: commit time and lock
Solr still respond to search queries during commit, only new indexations requests will have to wait (until end of commit?). So I don't think your users will experience increased response time during commits (unless your server is much undersized). Pierre -Message d'origine- De : Jonty Rhods [mailto:jonty.rh...@gmail.com] Envoyé : jeudi 21 juillet 2011 20:27 À : solr-user@lucene.apache.org Objet : Re: commit time and lock Actually i m worried about the response time. i k commiting around 500 docs in every 5 minutes. as i know,correct me if i m wrong; at the time of commiting solr server stop responding. my concern is how to minimize the response time so user not need to wait. or any other logic will require for my case. please suggest. regards jonty On Tuesday, June 21, 2011, Erick Erickson erickerick...@gmail.com wrote: What is it you want help with? You haven't told us what the problem you're trying to solve is. Are you asking how to speed up indexing? What have you tried? Have you looked at: http://wiki.apache.org/solr/FAQ#Performance? Best Erick On Tue, Jun 21, 2011 at 2:16 AM, Jonty Rhods jonty.rh...@gmail.com wrote: I am using solrj to index the data. I have around 5 docs indexed. As at the time of commit due to lock server stop giving response so I was calculating commit time: double starttemp = System.currentTimeMillis(); server.add(docs); server.commit(); System.out.println(total time in commit = + (System.currentTimeMillis() - starttemp)/1000); It taking around 9 second to commit the 5000 docs with 15 fields. However I am not confirm the lock time of index whether it is start since server.add(docs); time or server.commit(); time only. If I am changing from above to following server.add(docs); double starttemp = System.currentTimeMillis(); server.commit(); System.out.println(total time in commit = + (System.currentTimeMillis() - starttemp)/1000); then commit time becomes less then 1 second. I am not sure which one is right. please help. regards Jonty
RE: Solr 3.3: Exception in thread Lucene Merge Thread #1
I was wrong. After rebooting tomcat we discovered a new sweetness: /SEVERE: REFCOUNT ERROR: unreferenced org.apache.solr.core.SolrCore@3c753c75 (core.name) has a reference count of 1 22.07.2011 11:52:07 org.apache.solr.common.SolrException log SEVERE: java.lang.RuntimeException: java.io.IOException: Map failed at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1099) at org.apache.solr.core.SolrCore.init(SolrCore.java:585) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:463) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:316) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:207) at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:130) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:94) at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:273) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:254) at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:372) at org.apache.catalina.core.ApplicationFilterConfig.init(ApplicationFilterConfig.java:98) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4584) at org.apache.catalina.core.StandardContext$2.call(StandardContext.java:5262) at org.apache.catalina.core.StandardContext$2.call(StandardContext.java:5257) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314) at java.util.concurrent.FutureTask.run(FutureTask.java:149) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919) at java.lang.Thread.run(Thread.java:736) Caused by: java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:782) at org.apache.lucene.store.MMapDirectory$MMapIndexInput.init(MMapDirectory.java:264) at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216) at org.apache.lucene.index.FieldsReader.init(FieldsReader.java:129) at org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:244) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:92) at org.apache.lucene.index.DirectoryReader.init(DirectoryReader.java:113) at org.apache.lucene.index.ReadOnlyDirectoryReader.init(ReadOnlyDirectoryReader.java:29) at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:81) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:750) at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:75) at org.apache.lucene.index.IndexReader.open(IndexReader.java:428) at org.apache.lucene.index.IndexReader.open(IndexReader.java:371) at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1088) ... 18 more Caused by: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:779) ... 33 more/ Any ideas and/or suggestions? Best regards thank you, Sebastian -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-3-3-Exception-in-thread-Lucene-Merge-Thread-1-tp3185248p3190976.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: commit time and lock
Thanks for clarity. One more thing I want to know about optimization. Right now I am planning to optimize the server in 24 hour. Optimization is also time taking ( last time took around 13 minutes), so I want to know that : 1. when optimization is under process that time will solr server response or not? 2. if server will not response then how to do optimization of server fast or other way to do optimization so our user will not have to wait to finished optimization process. regards Jonty On Fri, Jul 22, 2011 at 2:44 PM, Pierre GOSSE pierre.go...@arisem.comwrote: Solr still respond to search queries during commit, only new indexations requests will have to wait (until end of commit?). So I don't think your users will experience increased response time during commits (unless your server is much undersized). Pierre -Message d'origine- De : Jonty Rhods [mailto:jonty.rh...@gmail.com] Envoyé : jeudi 21 juillet 2011 20:27 À : solr-user@lucene.apache.org Objet : Re: commit time and lock Actually i m worried about the response time. i k commiting around 500 docs in every 5 minutes. as i know,correct me if i m wrong; at the time of commiting solr server stop responding. my concern is how to minimize the response time so user not need to wait. or any other logic will require for my case. please suggest. regards jonty On Tuesday, June 21, 2011, Erick Erickson erickerick...@gmail.com wrote: What is it you want help with? You haven't told us what the problem you're trying to solve is. Are you asking how to speed up indexing? What have you tried? Have you looked at: http://wiki.apache.org/solr/FAQ#Performance? Best Erick On Tue, Jun 21, 2011 at 2:16 AM, Jonty Rhods jonty.rh...@gmail.com wrote: I am using solrj to index the data. I have around 5 docs indexed. As at the time of commit due to lock server stop giving response so I was calculating commit time: double starttemp = System.currentTimeMillis(); server.add(docs); server.commit(); System.out.println(total time in commit = + (System.currentTimeMillis() - starttemp)/1000); It taking around 9 second to commit the 5000 docs with 15 fields. However I am not confirm the lock time of index whether it is start since server.add(docs); time or server.commit(); time only. If I am changing from above to following server.add(docs); double starttemp = System.currentTimeMillis(); server.commit(); System.out.println(total time in commit = + (System.currentTimeMillis() - starttemp)/1000); then commit time becomes less then 1 second. I am not sure which one is right. please help. regards Jonty
convert date format at indexing time
Hi.I'm indexing binary documents as Word, pdf,... from a file system I'm extracting the attribute attr_creation_date for these documents but the format I'm getting is Wed Jan 14 12:13:00 CET 2004 instead 2004-01-14T12:12:00Z Is It possible to convert date format at indexing time? Thanks Best regards
RE: commit time and lock
Solr will response for search during optimization, but commits will have to wait the end of the optimization process. During optimization a new index is generated on disk by merging every single file of the current index into one big file, so you're server will be busy, especially regarding disk access. This may alter your response time and has very negative effect on the replication of index if you have a master/slave architecture. I've read here that optimization is not always a requirement to have an efficient index, due to some low level changes in lucene 3.xx, so maybe you don't really need optimization. What version of solr are you using ? Maybe someone can point toward a relevant link about optimization other than solr wiki http://wiki.apache.org/solr/SolrPerformanceFactors#Optimization_Considerations Pierre -Message d'origine- De : Jonty Rhods [mailto:jonty.rh...@gmail.com] Envoyé : vendredi 22 juillet 2011 12:45 À : solr-user@lucene.apache.org Objet : Re: commit time and lock Thanks for clarity. One more thing I want to know about optimization. Right now I am planning to optimize the server in 24 hour. Optimization is also time taking ( last time took around 13 minutes), so I want to know that : 1. when optimization is under process that time will solr server response or not? 2. if server will not response then how to do optimization of server fast or other way to do optimization so our user will not have to wait to finished optimization process. regards Jonty On Fri, Jul 22, 2011 at 2:44 PM, Pierre GOSSE pierre.go...@arisem.comwrote: Solr still respond to search queries during commit, only new indexations requests will have to wait (until end of commit?). So I don't think your users will experience increased response time during commits (unless your server is much undersized). Pierre -Message d'origine- De : Jonty Rhods [mailto:jonty.rh...@gmail.com] Envoyé : jeudi 21 juillet 2011 20:27 À : solr-user@lucene.apache.org Objet : Re: commit time and lock Actually i m worried about the response time. i k commiting around 500 docs in every 5 minutes. as i know,correct me if i m wrong; at the time of commiting solr server stop responding. my concern is how to minimize the response time so user not need to wait. or any other logic will require for my case. please suggest. regards jonty On Tuesday, June 21, 2011, Erick Erickson erickerick...@gmail.com wrote: What is it you want help with? You haven't told us what the problem you're trying to solve is. Are you asking how to speed up indexing? What have you tried? Have you looked at: http://wiki.apache.org/solr/FAQ#Performance? Best Erick On Tue, Jun 21, 2011 at 2:16 AM, Jonty Rhods jonty.rh...@gmail.com wrote: I am using solrj to index the data. I have around 5 docs indexed. As at the time of commit due to lock server stop giving response so I was calculating commit time: double starttemp = System.currentTimeMillis(); server.add(docs); server.commit(); System.out.println(total time in commit = + (System.currentTimeMillis() - starttemp)/1000); It taking around 9 second to commit the 5000 docs with 15 fields. However I am not confirm the lock time of index whether it is start since server.add(docs); time or server.commit(); time only. If I am changing from above to following server.add(docs); double starttemp = System.currentTimeMillis(); server.commit(); System.out.println(total time in commit = + (System.currentTimeMillis() - starttemp)/1000); then commit time becomes less then 1 second. I am not sure which one is right. please help. regards Jonty
Re: Geospatial queries in Solr
Ah, my mistake then. I will switch to using the geohash field. When doing my query I did run it against geohash but when I got Russia that was more incorrect than point so I stopped using it. Is there a timeline by which you expect the dateline issue to be addressed? I don't believe that will be an issue for me since I don't believe we have any information which will be from that area of the world but I'll have to verify. On Fri, Jul 22, 2011 at 3:35 AM, David Smiley (@MITRE.org) dsmi...@mitre.org wrote: Jamie, You are using the field named point which is based on PointFieldType. Keep in mind that just because this field type is named this way, does *not* mean at all that other fields don't hold points, or that this one is especially suited to it. Arguably this one is named poorly. This field type is loosely based on Solr's LatLonType but it doesn't yet fully implement capabilities like crossing the dateline and shapes other than a bounding box. I've enhanced the code to log a warning if the query shape is expanded to the bounding box. Any way, it's a currently a 2nd class citizen in LSP. I recommend you use the field named geohash which uses a field type name starting with Recursive Note that the same query you did with this field returned the polygon of Russia. There is an outstanding box in which we don't properly handle polygons crossing the dateline; other shapes are fine. ~ David - Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book -- View this message in context: http://lucene.472066.n3.nabble.com/Geospatial-queries-in-Solr-tp3183374p3190762.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Logically equivalent queries but vastly different no of results?
I think I know what it is. The second query has higher scores than the first. The additional condition domain_ids:(0^1.3 OR 1) which evaluates to true always - pushed up the scores and allows a LOT more records to pass. Is there a better way of doing this? Regards, Yee -- View this message in context: http://lucene.472066.n3.nabble.com/Logically-equivalent-queries-but-vastly-different-no-of-results-tp3190278p3191211.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Logically equivalent queries but vastly different no of results?
Am 22.07.2011 14:27, schrieb cnyee: I think I know what it is. The second query has higher scores than the first. The additional condition domain_ids:(0^1.3 OR 1) which evaluates to true always - pushed up the scores and allows a LOT more records to pass. This can't be, because the score doesn't affect the document count. Only relative score values are relevant, not absolute ones. Another guess: What's your query handler? If it's dismax, than the reason is quite obvious. -Kuli
Re: Solr 3.3: Exception in thread Lucene Merge Thread #1
IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 I'm confused why MMapDirectory is getting used with the IBM JVM... I had thought it would default to NIOFSDirectory on Linux w/ a non Oracle JVM. Are you specifically selecting MMapDirectory in solrconfig.xml? Can you try the Oracle JVM to see if that changes things? -Yonik http://www.lucidimagination.com On Fri, Jul 22, 2011 at 5:58 AM, mdz-munich sebastian.lu...@bsb-muenchen.de wrote: I was wrong. After rebooting tomcat we discovered a new sweetness: /SEVERE: REFCOUNT ERROR: unreferenced org.apache.solr.core.SolrCore@3c753c75 (core.name) has a reference count of 1 22.07.2011 11:52:07 org.apache.solr.common.SolrException log SEVERE: java.lang.RuntimeException: java.io.IOException: Map failed at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1099) at org.apache.solr.core.SolrCore.init(SolrCore.java:585) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:463) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:316) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:207) at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:130) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:94) at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:273) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:254) at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:372) at org.apache.catalina.core.ApplicationFilterConfig.init(ApplicationFilterConfig.java:98) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4584) at org.apache.catalina.core.StandardContext$2.call(StandardContext.java:5262) at org.apache.catalina.core.StandardContext$2.call(StandardContext.java:5257) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314) at java.util.concurrent.FutureTask.run(FutureTask.java:149) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919) at java.lang.Thread.run(Thread.java:736) Caused by: java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:782) at org.apache.lucene.store.MMapDirectory$MMapIndexInput.init(MMapDirectory.java:264) at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216) at org.apache.lucene.index.FieldsReader.init(FieldsReader.java:129) at org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:244) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:92) at org.apache.lucene.index.DirectoryReader.init(DirectoryReader.java:113) at org.apache.lucene.index.ReadOnlyDirectoryReader.init(ReadOnlyDirectoryReader.java:29) at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:81) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:750) at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:75) at org.apache.lucene.index.IndexReader.open(IndexReader.java:428) at org.apache.lucene.index.IndexReader.open(IndexReader.java:371) at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1088) ... 18 more Caused by: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:779) ... 33 more/ Any ideas and/or suggestions? Best regards thank you, Sebastian -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-3-3-Exception-in-thread-Lucene-Merge-Thread-1-tp3185248p3190976.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: commit time and lock
Hello, Pierre, can you tell us where you read that? I've read here that optimization is not always a requirement to have an efficient index, due to some low level changes in lucene 3.xx Marc. On Fri, Jul 22, 2011 at 2:10 PM, Pierre GOSSE pierre.go...@arisem.comwrote: Solr will response for search during optimization, but commits will have to wait the end of the optimization process. During optimization a new index is generated on disk by merging every single file of the current index into one big file, so you're server will be busy, especially regarding disk access. This may alter your response time and has very negative effect on the replication of index if you have a master/slave architecture. I've read here that optimization is not always a requirement to have an efficient index, due to some low level changes in lucene 3.xx, so maybe you don't really need optimization. What version of solr are you using ? Maybe someone can point toward a relevant link about optimization other than solr wiki http://wiki.apache.org/solr/SolrPerformanceFactors#Optimization_Considerations Pierre -Message d'origine- De : Jonty Rhods [mailto:jonty.rh...@gmail.com] Envoyé : vendredi 22 juillet 2011 12:45 À : solr-user@lucene.apache.org Objet : Re: commit time and lock Thanks for clarity. One more thing I want to know about optimization. Right now I am planning to optimize the server in 24 hour. Optimization is also time taking ( last time took around 13 minutes), so I want to know that : 1. when optimization is under process that time will solr server response or not? 2. if server will not response then how to do optimization of server fast or other way to do optimization so our user will not have to wait to finished optimization process. regards Jonty On Fri, Jul 22, 2011 at 2:44 PM, Pierre GOSSE pierre.go...@arisem.com wrote: Solr still respond to search queries during commit, only new indexations requests will have to wait (until end of commit?). So I don't think your users will experience increased response time during commits (unless your server is much undersized). Pierre -Message d'origine- De : Jonty Rhods [mailto:jonty.rh...@gmail.com] Envoyé : jeudi 21 juillet 2011 20:27 À : solr-user@lucene.apache.org Objet : Re: commit time and lock Actually i m worried about the response time. i k commiting around 500 docs in every 5 minutes. as i know,correct me if i m wrong; at the time of commiting solr server stop responding. my concern is how to minimize the response time so user not need to wait. or any other logic will require for my case. please suggest. regards jonty On Tuesday, June 21, 2011, Erick Erickson erickerick...@gmail.com wrote: What is it you want help with? You haven't told us what the problem you're trying to solve is. Are you asking how to speed up indexing? What have you tried? Have you looked at: http://wiki.apache.org/solr/FAQ#Performance? Best Erick On Tue, Jun 21, 2011 at 2:16 AM, Jonty Rhods jonty.rh...@gmail.com wrote: I am using solrj to index the data. I have around 5 docs indexed. As at the time of commit due to lock server stop giving response so I was calculating commit time: double starttemp = System.currentTimeMillis(); server.add(docs); server.commit(); System.out.println(total time in commit = + (System.currentTimeMillis() - starttemp)/1000); It taking around 9 second to commit the 5000 docs with 15 fields. However I am not confirm the lock time of index whether it is start since server.add(docs); time or server.commit(); time only. If I am changing from above to following server.add(docs); double starttemp = System.currentTimeMillis(); server.commit(); System.out.println(total time in commit = + (System.currentTimeMillis() - starttemp)/1000); then commit time becomes less then 1 second. I am not sure which one is right. please help. regards Jonty
Re: Updating fields in an existing document
Yes that's it if you add twice the same document (ie with the same id) it will replace it. On Thu, Jul 21, 2011 at 7:46 PM, Benson Margulies bimargul...@gmail.comwrote: A followup. The wiki has a whole discussion of the 'update' XML message. But solrj has nothing like it. Does that really exist? Is there a reason to use it? If I just 'add' the document a second time, it will replace? On Wed, Jul 20, 2011 at 7:08 PM, Jonathan Rochkind rochk...@jhu.edu wrote: Nope, you're not missing anything, there's no way to alter a document in an index but reindexing the whole document. Solr's architecture would make it difficult (although never say impossible) to do otherwise. But you're right it would be convenient for people other than you. Reindexing a single document ought not to be slow, although if you have many of them at once it could be, or if you end up needing to very frequently commit to an index it can indeed cause problems. From: Benson Margulies [bimargul...@gmail.com] Sent: Wednesday, July 20, 2011 6:05 PM To: solr-user Subject: Updating fields in an existing document We find ourselves in the following quandry: At initial index time, we store a value in a field, and we use it for facetting. So it, seemingly, has to be there as a field. However, from time to time, something happens that causes us to want to change this value. As far as we know, this requires us to completely re-index the document, which is slow. It struck me that we can't be the only people to go down this road, so I write to inquire if we are missing something.
Re: Solr 3.3: Exception in thread Lucene Merge Thread #1
On Fri, Jul 22, 2011 at 9:44 AM, Yonik Seeley yo...@lucidimagination.com wrote: IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 I'm confused why MMapDirectory is getting used with the IBM JVM... I had thought it would default to NIOFSDirectory on Linux w/ a non Oracle JVM. I verified that the MMapDirectory is selected by default with the IBM JVM (it must also contain the right Sun internal classes). Anyone else have experience with MMapDirectory w/ IBM's JVM? -Yonik http://www.lucidimagination.com Are you specifically selecting MMapDirectory in solrconfig.xml? Can you try the Oracle JVM to see if that changes things? -Yonik http://www.lucidimagination.com On Fri, Jul 22, 2011 at 5:58 AM, mdz-munich sebastian.lu...@bsb-muenchen.de wrote: I was wrong. After rebooting tomcat we discovered a new sweetness: /SEVERE: REFCOUNT ERROR: unreferenced org.apache.solr.core.SolrCore@3c753c75 (core.name) has a reference count of 1 22.07.2011 11:52:07 org.apache.solr.common.SolrException log SEVERE: java.lang.RuntimeException: java.io.IOException: Map failed at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1099) at org.apache.solr.core.SolrCore.init(SolrCore.java:585) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:463) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:316) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:207) at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:130) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:94) at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:273) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:254) at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:372) at org.apache.catalina.core.ApplicationFilterConfig.init(ApplicationFilterConfig.java:98) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4584) at org.apache.catalina.core.StandardContext$2.call(StandardContext.java:5262) at org.apache.catalina.core.StandardContext$2.call(StandardContext.java:5257) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314) at java.util.concurrent.FutureTask.run(FutureTask.java:149) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919) at java.lang.Thread.run(Thread.java:736) Caused by: java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:782) at org.apache.lucene.store.MMapDirectory$MMapIndexInput.init(MMapDirectory.java:264) at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216) at org.apache.lucene.index.FieldsReader.init(FieldsReader.java:129) at org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:244) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:92) at org.apache.lucene.index.DirectoryReader.init(DirectoryReader.java:113) at org.apache.lucene.index.ReadOnlyDirectoryReader.init(ReadOnlyDirectoryReader.java:29) at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:81) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:750) at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:75) at org.apache.lucene.index.IndexReader.open(IndexReader.java:428) at org.apache.lucene.index.IndexReader.open(IndexReader.java:371) at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1088) ... 18 more Caused by: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:779) ... 33 more/ Any ideas and/or suggestions? Best regards thank you, Sebastian -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-3-3-Exception-in-thread-Lucene-Merge-Thread-1-tp3185248p3190976.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Geospatial queries in Solr
Wrapping the dateline or being able to encircle one of the poles (but not necessarily both) are polygon query features that I feel need to be addressed before this module is first released (whenever that is), definitely. And arguably before benchmarking, which we're looking to focus on soon. So Soon. W/i 2 months? Just a guess. ~ David On Jul 22, 2011, at 8:18 AM, Jamie Johnson wrote: Ah, my mistake then. I will switch to using the geohash field. When doing my query I did run it against geohash but when I got Russia that was more incorrect than point so I stopped using it. Is there a timeline by which you expect the dateline issue to be addressed? I don't believe that will be an issue for me since I don't believe we have any information which will be from that area of the world but I'll have to verify. On Fri, Jul 22, 2011 at 3:35 AM, David Smiley (@MITRE.org) dsmi...@mitre.org wrote: Jamie, You are using the field named point which is based on PointFieldType. Keep in mind that just because this field type is named this way, does *not* mean at all that other fields don't hold points, or that this one is especially suited to it. Arguably this one is named poorly. This field type is loosely based on Solr's LatLonType but it doesn't yet fully implement capabilities like crossing the dateline and shapes other than a bounding box. I've enhanced the code to log a warning if the query shape is expanded to the bounding box. Any way, it's a currently a 2nd class citizen in LSP. I recommend you use the field named geohash which uses a field type name starting with Recursive Note that the same query you did with this field returned the polygon of Russia. There is an outstanding box in which we don't properly handle polygons crossing the dateline; other shapes are fine. ~ David - Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book -- View this message in context: http://lucene.472066.n3.nabble.com/Geospatial-queries-in-Solr-tp3183374p3190762.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Geospatial queries in Solr
Thanks David. I'm going to continue to play with this, as an FYI you were spot on, changing to use a geohash field worked with the previous test. Again I appreciate all of the information, and awesome work. On Fri, Jul 22, 2011 at 10:05 AM, Smiley, David W. dsmi...@mitre.org wrote: Wrapping the dateline or being able to encircle one of the poles (but not necessarily both) are polygon query features that I feel need to be addressed before this module is first released (whenever that is), definitely. And arguably before benchmarking, which we're looking to focus on soon. So Soon. W/i 2 months? Just a guess. ~ David On Jul 22, 2011, at 8:18 AM, Jamie Johnson wrote: Ah, my mistake then. I will switch to using the geohash field. When doing my query I did run it against geohash but when I got Russia that was more incorrect than point so I stopped using it. Is there a timeline by which you expect the dateline issue to be addressed? I don't believe that will be an issue for me since I don't believe we have any information which will be from that area of the world but I'll have to verify. On Fri, Jul 22, 2011 at 3:35 AM, David Smiley (@MITRE.org) dsmi...@mitre.org wrote: Jamie, You are using the field named point which is based on PointFieldType. Keep in mind that just because this field type is named this way, does *not* mean at all that other fields don't hold points, or that this one is especially suited to it. Arguably this one is named poorly. This field type is loosely based on Solr's LatLonType but it doesn't yet fully implement capabilities like crossing the dateline and shapes other than a bounding box. I've enhanced the code to log a warning if the query shape is expanded to the bounding box. Any way, it's a currently a 2nd class citizen in LSP. I recommend you use the field named geohash which uses a field type name starting with Recursive Note that the same query you did with this field returned the polygon of Russia. There is an outstanding box in which we don't properly handle polygons crossing the dateline; other shapes are fine. ~ David - Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book -- View this message in context: http://lucene.472066.n3.nabble.com/Geospatial-queries-in-Solr-tp3183374p3190762.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: commit time and lock
Hi Mark I've read that in a thread title Weird optimize performance degradation, where Erick Erickson states that Older versions of Lucene would search faster on an optimized index, but this is no longer necessary., and more recently in a thread you initiated a month ago Question about optimization. I'll also be very interested if anyone had a more precise idea/datas of benefits and tradeoff of optimize vs merge ... Pierre -Message d'origine- De : Marc SCHNEIDER [mailto:marc.schneide...@gmail.com] Envoyé : vendredi 22 juillet 2011 15:45 À : solr-user@lucene.apache.org Objet : Re: commit time and lock Hello, Pierre, can you tell us where you read that? I've read here that optimization is not always a requirement to have an efficient index, due to some low level changes in lucene 3.xx Marc. On Fri, Jul 22, 2011 at 2:10 PM, Pierre GOSSE pierre.go...@arisem.comwrote: Solr will response for search during optimization, but commits will have to wait the end of the optimization process. During optimization a new index is generated on disk by merging every single file of the current index into one big file, so you're server will be busy, especially regarding disk access. This may alter your response time and has very negative effect on the replication of index if you have a master/slave architecture. I've read here that optimization is not always a requirement to have an efficient index, due to some low level changes in lucene 3.xx, so maybe you don't really need optimization. What version of solr are you using ? Maybe someone can point toward a relevant link about optimization other than solr wiki http://wiki.apache.org/solr/SolrPerformanceFactors#Optimization_Considerations Pierre -Message d'origine- De : Jonty Rhods [mailto:jonty.rh...@gmail.com] Envoyé : vendredi 22 juillet 2011 12:45 À : solr-user@lucene.apache.org Objet : Re: commit time and lock Thanks for clarity. One more thing I want to know about optimization. Right now I am planning to optimize the server in 24 hour. Optimization is also time taking ( last time took around 13 minutes), so I want to know that : 1. when optimization is under process that time will solr server response or not? 2. if server will not response then how to do optimization of server fast or other way to do optimization so our user will not have to wait to finished optimization process. regards Jonty On Fri, Jul 22, 2011 at 2:44 PM, Pierre GOSSE pierre.go...@arisem.com wrote: Solr still respond to search queries during commit, only new indexations requests will have to wait (until end of commit?). So I don't think your users will experience increased response time during commits (unless your server is much undersized). Pierre -Message d'origine- De : Jonty Rhods [mailto:jonty.rh...@gmail.com] Envoyé : jeudi 21 juillet 2011 20:27 À : solr-user@lucene.apache.org Objet : Re: commit time and lock Actually i m worried about the response time. i k commiting around 500 docs in every 5 minutes. as i know,correct me if i m wrong; at the time of commiting solr server stop responding. my concern is how to minimize the response time so user not need to wait. or any other logic will require for my case. please suggest. regards jonty On Tuesday, June 21, 2011, Erick Erickson erickerick...@gmail.com wrote: What is it you want help with? You haven't told us what the problem you're trying to solve is. Are you asking how to speed up indexing? What have you tried? Have you looked at: http://wiki.apache.org/solr/FAQ#Performance? Best Erick On Tue, Jun 21, 2011 at 2:16 AM, Jonty Rhods jonty.rh...@gmail.com wrote: I am using solrj to index the data. I have around 5 docs indexed. As at the time of commit due to lock server stop giving response so I was calculating commit time: double starttemp = System.currentTimeMillis(); server.add(docs); server.commit(); System.out.println(total time in commit = + (System.currentTimeMillis() - starttemp)/1000); It taking around 9 second to commit the 5000 docs with 15 fields. However I am not confirm the lock time of index whether it is start since server.add(docs); time or server.commit(); time only. If I am changing from above to following server.add(docs); double starttemp = System.currentTimeMillis(); server.commit(); System.out.println(total time in commit = + (System.currentTimeMillis() - starttemp)/1000); then commit time becomes less then 1 second. I am not sure which one is right. please help. regards Jonty
problem searching on non standard characters
How does one search for words with characters like # and +. I have tried searching solr with #test and \#test but all my results always come up with test and not #test. Is this some kind of configuration option I need to set in solr? -- - sent from my mobile 6176064373
Re: commit time and lock
On 7/22/2011 8:23 AM, Pierre GOSSE wrote: I've read that in a thread title Weird optimize performance degradation, where Erick Erickson states that Older versions of Lucene would search faster on an optimized index, but this is no longer necessary., and more recently in a thread you initiated a month ago Question about optimization. I'll also be very interested if anyone had a more precise idea/datas of benefits and tradeoff of optimize vs merge ... My most recent testing has been with Solr 3.2.0. I have noticed some speedup after optimizing an index, but the gain is not earth-shattering. My index consists of 7 shards. One of them is small, and receives all new documents every two minutes. The others are large, and aside from deletes, are mostly static. Once a day, the oldest data is distributed from the small shard to its proper place in the other six shards. The small shard is optimized once an hour, and usually takes less than a minute. I optimize one large shard every day, so each one gets optimized once every six days. That optimize takes 10-15 minutes. The only reason that I optimize is to remove deleted documents, whatever speedup I get is just icing on the cake. Deleted documents take up space and continue to influence the relevance scoring of queries, so I want to remove them. Thanks, Shawn
Re: problem searching on non standard characters
Check your analyzers to make sure that these characters are not getting stripped out in the tokenization process, the url for 3.3 is somewhere along the lines of: http://localhost/solr/admin/analysis.jsp?highlight=on And you should be indeed be searching on \#test. François On Jul 22, 2011, at 10:34 AM, Jason Toy wrote: How does one search for words with characters like # and +. I have tried searching solr with #test and \#test but all my results always come up with test and not #test. Is this some kind of configuration option I need to set in solr? -- - sent from my mobile 6176064373
Re: problem searching on non standard characters
On 7/22/2011 8:34 AM, Jason Toy wrote: How does one search for words with characters like # and +. I have tried searching solr with #test and \#test but all my results always come up with test and not #test. Is this some kind of configuration option I need to set in solr? I would guess that your analysis chain (in schema.xml) includes something that removes and/or splits terms at non-alphanumeric characters. There are a several components that do this, but WordDelimiterFilter is the one that comes to mind most readily. I've never used the StandardTokenizer, but I believe it might do something similar. Thanks, Shawn
Re: problem searching on non standard characters
Adding to my previous reply, I just did a quick check on the 'text_en' and 'text_en_splitting' field types and they both strip leading '#'. Cheers François On Jul 22, 2011, at 10:49 AM, Shawn Heisey wrote: On 7/22/2011 8:34 AM, Jason Toy wrote: How does one search for words with characters like # and +. I have tried searching solr with #test and \#test but all my results always come up with test and not #test. Is this some kind of configuration option I need to set in solr? I would guess that your analysis chain (in schema.xml) includes something that removes and/or splits terms at non-alphanumeric characters. There are a several components that do this, but WordDelimiterFilter is the one that comes to mind most readily. I've never used the StandardTokenizer, but I believe it might do something similar. Thanks, Shawn
RE: Re: previous and next rows of current record
Yes exactly same problem i m facing. Is there any way to resolve this issue.. I am not sure what you mean, any way to resolve this issue. Did you read and understand what I wrote below? I have nothing more to add. What is it you're looking for? The way to provide that kind of next/previous is to write application code to do it. Although it's not easy to do cleanly a web app because of the sessionless architecture of the web. What are you using for your client application? But honestly I probably have nothing more to say on the topic. From : Jonathan Rochkind To : solr-user@lucene.apache.org; Subject : Re: previous and next rows of current record I think maybe I know what you mean. You have a result set generated by a query. You have an item detail page in your web app -- on that item detail page, you want to give next/previous buttons for current search results. If that's it, read on (although news isn't good), if that's not it, ignore me. There is no good way to do it. Although it's not really so much a solr problem. As far as Solr is concerned, if you know the query, and you know the current row into the query i, then just ask Solr for rows=1start=$(i=1) to get previous, or i+1 to get next. (You can't send $(i-1) or $(i+1) to Solr that's just short hand, your app would have to calculate em and send the literals). The problem is architecting a web app so when you are on an item detail page, the app knows what the current Solr query was, and what the i index into it was. The app I work on wants to provide this feature too, but I am so unhappy with what it currently does (it is both ugly AND does not actually work at all right on several very common cases), that I am definitely not going to provide it as an example. But if you are willing to have your web app send the current search and the index in the URL to the item detail page, that'd certainly make it easier. It's not so much a Solr problem -- the answer in Solr is pretty clear. Keep track of what index into your results you are on, and then just ask for one previous or more. But there's no great way to make a web app taht actually does that without horrid urls. There's nothing built into Solr to help you. Solr is pretty much sessionless/stateless, it's got no idea what the 'current' search for your particular session is. On 7/21/2011 2:38 PM, Bob Sandiford wrote: But - what is it that makes '9' the next id after '5'? why not '6'? Or '91238412'? or '4'? i.e. you still haven't answered the question about what 'next' and 'previous' really means... But - if you already know that '9' is the next page, why not just do another query with id '9' to get the next record? Bob Sandiford | Lead Software Engineer | SirsiDynix P: 800.288.8020 X6943 | bob.sandif...@sirsidynix.com www.sirsidynix.com -Original Message- From: Jonty Rhods [mailto:jonty.rh...@gmail.com] Sent: Thursday, July 21, 2011 2:33 PM To: solr-user@lucene.apache.org Subject: Re: previous and next rows of current record Hi in my case there is no id sequence. id is generated sequence wise for all category. but when we filter by category then same id become random. If i m on detail page which have id 5 and nrxt id is 9 so on same page my requirment is to get next id is 9. On Thursday, July 21, 2011, Bob Sandiford wrote: Well, it sort of depends on what you mean by the 'previous' and the 'next' record. Do you have some type of sequencing built into your concept of your solr / lucene indexes? Do you have sequential id's? i.e. What's the use case, and what's the data available to support your use case? Bob Sandiford | Lead Software Engineer | SirsiDynix P: 800.288.8020 X6943 | bob.sandif...@sirsidynix.com www.sirsidynix.com -Original Message- From: Jonty Rhods [mailto:jonty.rh...@gmail.com] Sent: Thursday, July 21, 2011 2:18 PM To: solr-user@lucene.apache.org Subject: Re: previous and next rows of current record Pls help.. On Thursday, July 21, 2011, Jonty Rhods wrote: Hi, Is there any special query in solr to get the previous and next record of current record.I am getting single record detail using id from solr server. I need to get next and previous on detail page. regardsJonty
RE: commit time and lock
Merging does not happen often enough to keep deleted documents to a low enough count ? Maybe there's a need to have partial optimization available in solr, meaning that segment with too much deleted document could be copied to a new file without unnecessary datas. That way cleaning deleted datas could be compatible with having light replications. I'm worried by this idea of deleted documents influencing relevance scores, any pointer to how important this influence may be ? Pierre -Message d'origine- De : Shawn Heisey [mailto:s...@elyograg.org] Envoyé : vendredi 22 juillet 2011 16:42 À : solr-user@lucene.apache.org Objet : Re: commit time and lock On 7/22/2011 8:23 AM, Pierre GOSSE wrote: I've read that in a thread title Weird optimize performance degradation, where Erick Erickson states that Older versions of Lucene would search faster on an optimized index, but this is no longer necessary., and more recently in a thread you initiated a month ago Question about optimization. I'll also be very interested if anyone had a more precise idea/datas of benefits and tradeoff of optimize vs merge ... My most recent testing has been with Solr 3.2.0. I have noticed some speedup after optimizing an index, but the gain is not earth-shattering. My index consists of 7 shards. One of them is small, and receives all new documents every two minutes. The others are large, and aside from deletes, are mostly static. Once a day, the oldest data is distributed from the small shard to its proper place in the other six shards. The small shard is optimized once an hour, and usually takes less than a minute. I optimize one large shard every day, so each one gets optimized once every six days. That optimize takes 10-15 minutes. The only reason that I optimize is to remove deleted documents, whatever speedup I get is just icing on the cake. Deleted documents take up space and continue to influence the relevance scoring of queries, so I want to remove them. Thanks, Shawn
RE: commit time and lock
How old is 'older'? I'm pretty sure I'm still getting much faster performance on an optimized index in Solr 1.4. This could be due to the nature of my index and queries (which include some medium sized stored fields, and extensive facetting -- facetting on up to a dozen fields in every request, where each field can include millions of unique values. Amazing I can do this with good performance at all!). It's also possible i'm wrong about that faster performance, i haven't done robustly valid benchmarking on a clone of my production index yet. But it really looks like that way to me, from what investigation I have done. If the answer is that optimization is believed no longer neccesary on versions LATER than 1.4, that might be the simplest explanation. From: Pierre GOSSE [pierre.go...@arisem.com] Sent: Friday, July 22, 2011 10:23 AM To: solr-user@lucene.apache.org Subject: RE: commit time and lock Hi Mark I've read that in a thread title Weird optimize performance degradation, where Erick Erickson states that Older versions of Lucene would search faster on an optimized index, but this is no longer necessary., and more recently in a thread you initiated a month ago Question about optimization. I'll also be very interested if anyone had a more precise idea/datas of benefits and tradeoff of optimize vs merge ... Pierre -Message d'origine- De : Marc SCHNEIDER [mailto:marc.schneide...@gmail.com] Envoyé : vendredi 22 juillet 2011 15:45 À : solr-user@lucene.apache.org Objet : Re: commit time and lock Hello, Pierre, can you tell us where you read that? I've read here that optimization is not always a requirement to have an efficient index, due to some low level changes in lucene 3.xx Marc. On Fri, Jul 22, 2011 at 2:10 PM, Pierre GOSSE pierre.go...@arisem.comwrote: Solr will response for search during optimization, but commits will have to wait the end of the optimization process. During optimization a new index is generated on disk by merging every single file of the current index into one big file, so you're server will be busy, especially regarding disk access. This may alter your response time and has very negative effect on the replication of index if you have a master/slave architecture. I've read here that optimization is not always a requirement to have an efficient index, due to some low level changes in lucene 3.xx, so maybe you don't really need optimization. What version of solr are you using ? Maybe someone can point toward a relevant link about optimization other than solr wiki http://wiki.apache.org/solr/SolrPerformanceFactors#Optimization_Considerations Pierre -Message d'origine- De : Jonty Rhods [mailto:jonty.rh...@gmail.com] Envoyé : vendredi 22 juillet 2011 12:45 À : solr-user@lucene.apache.org Objet : Re: commit time and lock Thanks for clarity. One more thing I want to know about optimization. Right now I am planning to optimize the server in 24 hour. Optimization is also time taking ( last time took around 13 minutes), so I want to know that : 1. when optimization is under process that time will solr server response or not? 2. if server will not response then how to do optimization of server fast or other way to do optimization so our user will not have to wait to finished optimization process. regards Jonty On Fri, Jul 22, 2011 at 2:44 PM, Pierre GOSSE pierre.go...@arisem.com wrote: Solr still respond to search queries during commit, only new indexations requests will have to wait (until end of commit?). So I don't think your users will experience increased response time during commits (unless your server is much undersized). Pierre -Message d'origine- De : Jonty Rhods [mailto:jonty.rh...@gmail.com] Envoyé : jeudi 21 juillet 2011 20:27 À : solr-user@lucene.apache.org Objet : Re: commit time and lock Actually i m worried about the response time. i k commiting around 500 docs in every 5 minutes. as i know,correct me if i m wrong; at the time of commiting solr server stop responding. my concern is how to minimize the response time so user not need to wait. or any other logic will require for my case. please suggest. regards jonty On Tuesday, June 21, 2011, Erick Erickson erickerick...@gmail.com wrote: What is it you want help with? You haven't told us what the problem you're trying to solve is. Are you asking how to speed up indexing? What have you tried? Have you looked at: http://wiki.apache.org/solr/FAQ#Performance? Best Erick On Tue, Jun 21, 2011 at 2:16 AM, Jonty Rhods jonty.rh...@gmail.com wrote: I am using solrj to index the data. I have around 5 docs indexed. As at the time of commit due to lock server stop giving response so I was calculating commit time: double starttemp = System.currentTimeMillis(); server.add(docs); server.commit();
Re: commit time and lock
On 7/22/2011 9:32 AM, Pierre GOSSE wrote: Merging does not happen often enough to keep deleted documents to a low enough count ? Maybe there's a need to have partial optimization available in solr, meaning that segment with too much deleted document could be copied to a new file without unnecessary datas. That way cleaning deleted datas could be compatible with having light replications. I'm worried by this idea of deleted documents influencing relevance scores, any pointer to how important this influence may be ? I've got a pretty high mergeFactor, for fast indexing. Also, I want to know for sure and control when merges happen, so I am not leaving it up to Lucene/Solr. Right now the largest number of deleted documents on any shard at this moment is 45347. The shard (17.65GB) contains 9663271 documents, in six segments. That will be one HUGE segment (from the last optimize) and five very very tiny segments, each with only a few thousand documents in them. Tonight when the document distribution process runs, that index will be optimized again. Tomorrow night a different shard will be optimized. Deleted documents can (and do) happen anywhere in the index, so even if I had a lot of largish segments rather than one huge segment, it's very likely that just expunging deletes would still result in the entire index being merged, so I am not losing anything by doing a full optimize, and I am gaining a small bit of performance. The 45000 deletes mentioned above represent less than half a percent of the shard, so the influence on relevance is *probably* not large ... but that's not something I can say definitively. I think it all depends on what people are searching for and how common the terms in the deleted documents are. Thanks, Shawn
Re: Solr 3.3: Exception in thread Lucene Merge Thread #1
Hi Yonik,
thanks for your reply!
Are you specifically selecting MMapDirectory in solrconfig.xml?
Nope.
We installed Oracle's Runtime from
http://java.com/de/download/linux_manual.jsp?locale=de
/java.runtime.name = Java(TM) SE Runtime Environment
sun.boot.library.path = /usr/java/jdk1.6.0_26/jre/lib/amd64
java.vm.version = 20.1-b02
shared.loader =
java.vm.vendor = Sun Microsystems Inc.
enable.master = true
java.vendor.url = http://java.sun.com/
path.separator = :
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
tomcat.util.buf.StringCache.byte.enabled = true
file.encoding.pkg = sun.io
java.util.logging.config.file =
/local/master01_tomcat7x_solr33x/conf/logging.properties
user.country = DE
sun.java.launcher = SUN_STANDARD
sun.os.patch.level = unknown
java.vm.specification.name = Java Virtual Machine Specification
user.dir = /local/master01_tomcat7x_solr33x/logs
solr.abortOnConfigurationError = true
java.runtime.version = 1.6.0_26-b03
java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
java.endorsed.dirs = /local/master01_tomcat7x_solr33x/endorsed
os.arch = amd64
java.io.tmpdir = /local/master01_tomcat7x_solr33x/temp
line.separator = /
But no success with 1000 docs/batch, this was thrown during optimize:
/
22.07.2011 18:44:05 org.apache.solr.core.SolrCore execute
INFO: [core.digi20] webapp=/solr path=/update params={} status=500
QTime=87540
22.07.2011 18:44:05 org.apache.solr.common.SolrException log
SEVERE: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
at
org.apache.lucene.store.MMapDirectory$MMapIndexInput.init(MMapDirectory.java:303)
at
org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:217)
at org.apache.lucene.index.FieldsReader.init(FieldsReader.java:129)
at
org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:245)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:117)
at
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:703)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4196)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3863)
at
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2715)
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2525)
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2462)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:410)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
at
org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:154)
at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:177)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:77)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:67)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:240)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:164)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:563)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:403)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:301)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:162)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:309)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: Map failed
Re: Solr 3.3: Exception in thread Lucene Merge Thread #1
OK, best guess is that you're going over some per-process address space limit.
Try seeing what ulimit -a says.
-Yonik
http://www.lucidimagination.com
On Fri, Jul 22, 2011 at 12:51 PM, mdz-munich
sebastian.lu...@bsb-muenchen.de wrote:
Hi Yonik,
thanks for your reply!
Are you specifically selecting MMapDirectory in solrconfig.xml?
Nope.
We installed Oracle's Runtime from
http://java.com/de/download/linux_manual.jsp?locale=de
/java.runtime.name = Java(TM) SE Runtime Environment
sun.boot.library.path = /usr/java/jdk1.6.0_26/jre/lib/amd64
java.vm.version = 20.1-b02
shared.loader =
java.vm.vendor = Sun Microsystems Inc.
enable.master = true
java.vendor.url = http://java.sun.com/
path.separator = :
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
tomcat.util.buf.StringCache.byte.enabled = true
file.encoding.pkg = sun.io
java.util.logging.config.file =
/local/master01_tomcat7x_solr33x/conf/logging.properties
user.country = DE
sun.java.launcher = SUN_STANDARD
sun.os.patch.level = unknown
java.vm.specification.name = Java Virtual Machine Specification
user.dir = /local/master01_tomcat7x_solr33x/logs
solr.abortOnConfigurationError = true
java.runtime.version = 1.6.0_26-b03
java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
java.endorsed.dirs = /local/master01_tomcat7x_solr33x/endorsed
os.arch = amd64
java.io.tmpdir = /local/master01_tomcat7x_solr33x/temp
line.separator = /
But no success with 1000 docs/batch, this was thrown during optimize:
/
22.07.2011 18:44:05 org.apache.solr.core.SolrCore execute
INFO: [core.digi20] webapp=/solr path=/update params={} status=500
QTime=87540
22.07.2011 18:44:05 org.apache.solr.common.SolrException log
SEVERE: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
at
org.apache.lucene.store.MMapDirectory$MMapIndexInput.init(MMapDirectory.java:303)
at
org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:217)
at org.apache.lucene.index.FieldsReader.init(FieldsReader.java:129)
at
org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:245)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:117)
at
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:703)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4196)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3863)
at
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
at
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2715)
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2525)
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2462)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:410)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
at
org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:154)
at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:177)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:77)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:67)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:240)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:164)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:563)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:403)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:301)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:162)
at
Re: Solr 3.3: Exception in thread Lucene Merge Thread #1
It says:
/core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 257869
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) 28063940
open files (-n) 8192
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 257869
virtual memory (kbytes, -v) 27216080
file locks (-x) unlimited/
Best regards,
Sebastian
Yonik Seeley-2-2 wrote:
OK, best guess is that you're going over some per-process address space
limit.
Try seeing what ulimit -a says.
-Yonik
http://www.lucidimagination.com
On Fri, Jul 22, 2011 at 12:51 PM, mdz-munich
lt;sebastian.lu...@bsb-muenchen.degt; wrote:
Hi Yonik,
thanks for your reply!
Are you specifically selecting MMapDirectory in solrconfig.xml?
Nope.
We installed Oracle's Runtime from
http://java.com/de/download/linux_manual.jsp?locale=de
/java.runtime.name = Java(TM) SE Runtime Environment
sun.boot.library.path = /usr/java/jdk1.6.0_26/jre/lib/amd64
java.vm.version = 20.1-b02
shared.loader =
java.vm.vendor = Sun Microsystems Inc.
enable.master = true
java.vendor.url = http://java.sun.com/
path.separator = :
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
tomcat.util.buf.StringCache.byte.enabled = true
file.encoding.pkg = sun.io
java.util.logging.config.file =
/local/master01_tomcat7x_solr33x/conf/logging.properties
user.country = DE
sun.java.launcher = SUN_STANDARD
sun.os.patch.level = unknown
java.vm.specification.name = Java Virtual Machine Specification
user.dir = /local/master01_tomcat7x_solr33x/logs
solr.abortOnConfigurationError = true
java.runtime.version = 1.6.0_26-b03
java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
java.endorsed.dirs = /local/master01_tomcat7x_solr33x/endorsed
os.arch = amd64
java.io.tmpdir = /local/master01_tomcat7x_solr33x/temp
line.separator = /
But no success with 1000 docs/batch, this was thrown during optimize:
/
22.07.2011 18:44:05 org.apache.solr.core.SolrCore execute
INFO: [core.digi20] webapp=/solr path=/update params={} status=500
QTime=87540
22.07.2011 18:44:05 org.apache.solr.common.SolrException log
SEVERE: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
at
org.apache.lucene.store.MMapDirectory$MMapIndexInput.init(MMapDirectory.java:303)
at
org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:217)
at
org.apache.lucene.index.FieldsReader.init(FieldsReader.java:129)
at
org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:245)
at
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:117)
at
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:703)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4196)
at
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3863)
at
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
at
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2715)
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2525)
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2462)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:410)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
at
org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:154)
at
org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:177)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:77)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:67)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:240)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:164)
at
Re: Solr 3.3: Exception in thread Lucene Merge Thread #1
Yep, there ya go... your OS configuration is limiting you to 27G of
virtual address space per process. Consider setting that to
unlimited.
-Yonik
http://www.lucidimagination.com
On Fri, Jul 22, 2011 at 1:05 PM, mdz-munich
sebastian.lu...@bsb-muenchen.de wrote:
It says:
/core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 257869
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) 28063940
open files (-n) 8192
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 257869
virtual memory (kbytes, -v) 27216080
file locks (-x) unlimited/
Best regards,
Sebastian
Yonik Seeley-2-2 wrote:
OK, best guess is that you're going over some per-process address space
limit.
Try seeing what ulimit -a says.
-Yonik
http://www.lucidimagination.com
On Fri, Jul 22, 2011 at 12:51 PM, mdz-munich
lt;sebastian.lu...@bsb-muenchen.degt; wrote:
Hi Yonik,
thanks for your reply!
Are you specifically selecting MMapDirectory in solrconfig.xml?
Nope.
We installed Oracle's Runtime from
http://java.com/de/download/linux_manual.jsp?locale=de
/java.runtime.name = Java(TM) SE Runtime Environment
sun.boot.library.path = /usr/java/jdk1.6.0_26/jre/lib/amd64
java.vm.version = 20.1-b02
shared.loader =
java.vm.vendor = Sun Microsystems Inc.
enable.master = true
java.vendor.url = http://java.sun.com/
path.separator = :
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
tomcat.util.buf.StringCache.byte.enabled = true
file.encoding.pkg = sun.io
java.util.logging.config.file =
/local/master01_tomcat7x_solr33x/conf/logging.properties
user.country = DE
sun.java.launcher = SUN_STANDARD
sun.os.patch.level = unknown
java.vm.specification.name = Java Virtual Machine Specification
user.dir = /local/master01_tomcat7x_solr33x/logs
solr.abortOnConfigurationError = true
java.runtime.version = 1.6.0_26-b03
java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
java.endorsed.dirs = /local/master01_tomcat7x_solr33x/endorsed
os.arch = amd64
java.io.tmpdir = /local/master01_tomcat7x_solr33x/temp
line.separator = /
But no success with 1000 docs/batch, this was thrown during optimize:
/
22.07.2011 18:44:05 org.apache.solr.core.SolrCore execute
INFO: [core.digi20] webapp=/solr path=/update params={} status=500
QTime=87540
22.07.2011 18:44:05 org.apache.solr.common.SolrException log
SEVERE: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
at
org.apache.lucene.store.MMapDirectory$MMapIndexInput.init(MMapDirectory.java:303)
at
org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:217)
at
org.apache.lucene.index.FieldsReader.init(FieldsReader.java:129)
at
org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:245)
at
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:117)
at
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:703)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4196)
at
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3863)
at
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
at
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2715)
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2525)
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2462)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:410)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
at
org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:154)
at
org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:177)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:77)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:67)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
Re: Solr 3.3: Exception in thread Lucene Merge Thread #1
Maybe it's important: - The OS (Open Suse 10) is virtualized on VMWare - Network Attached Storage Best regards Sebastian -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-3-3-Exception-in-thread-Lucene-Merge-Thread-1-tp3185248p3191986.html Sent from the Solr - User mailing list archive at Nabble.com.
Rounding errors in solr
Hi all, I've noticed some peculiar scoring issues going on in my application. For example, I have a field that is multivalued and has several records that have the same value. For example, arr name=references strNational Society of Animal Lovers/str strNat. Soc. of Ani. Lov./str /arr I have about 300 records with that exact value. Now, when I do a search for references:(national society animal lovers), I get the following results: id252/id id159/id id82/id id452/id id105/id When I do a search for references:(nat soc ani lov), I get the results ordered differently: id510/id id122/id id501/id id82/id id252/id When I load all the records that match, I notice that at some point, the scores aren't the same but differ by only a little: 1.471928 in one and the one before it was 1.471929 I turned on debugQuery=on and the scores for each of those two records are exactly the same. Therefore, I think there is some kind of rounding error going on. Is there a way I can fix this? Alternatively, can I sort by a rounded version of the score? I tried sort=round(score,5) but I get the following message: Can't determine Sort Order: 'round(score,5) ', pos=5 I also tried sort=sum(score,1) just to see if I was using round incorrectly but I get an error message there too saying score is not a recognized field. Please help! Thanks, Brian Lamb
Re: Rounding errors in solr
On Fri, Jul 22, 2011 at 4:11 PM, Brian Lamb brian.l...@journalexperts.com wrote: I've noticed some peculiar scoring issues going on in my application. For example, I have a field that is multivalued and has several records that have the same value. For example, arr name=references strNational Society of Animal Lovers/str strNat. Soc. of Ani. Lov./str /arr I have about 300 records with that exact value. Now, when I do a search for references:(national society animal lovers), I get the following results: id252/id id159/id id82/id id452/id id105/id When I do a search for references:(nat soc ani lov), I get the results ordered differently: id510/id id122/id id501/id id82/id id252/id When I load all the records that match, I notice that at some point, the scores aren't the same but differ by only a little: 1.471928 in one and the one before it was 1.471929 32 bit floats only have 7 decimal digits of precision, and in floating point land (a+b+c) can be slightly different than (c+b+a) -Yonik http://www.lucidimagination.com
saving timestamps in trunk broken?
In Solr 1.3.1 I am able to store timestamps in my docs so that I query them.
In trunk when I try to store a doc with a timestamp I get a sever error, is
there a different way I should store this data or is this a bug?
Jul 22, 2011 7:20:14 PM org.apache.solr.update.processor.LogUpdateProcessor
finish
INFO: {add=[Post 75004824785129473]} 0 0
Jul 22, 2011 7:20:14 PM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: ERROR: [doc=Post
75004824785129473] Error adding field 'at_d'='2011-05-30T01:05:18Z'
at
org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:300)
at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:60)
at
org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:115)
at
org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:127)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:77)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:67)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1401)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:353)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:248)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.lang.NumberFormatException: For input string:
2011-05-30T01:05:18Z
at
sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1222)
at java.lang.Double.parseDouble(Double.java:510)
at org.apache.solr.schema.TrieField.createField(TrieField.java:509)
at
org.apache.solr.schema.SchemaField.createField(SchemaField.java:107)
at
org.apache.solr.update.DocumentBuilder.addField(DocumentBuilder.java:202)
at
org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:261)
--
- sent from my mobile
6176064373
Re: saving timestamps in trunk broken?
: In Solr 1.3.1 I am able to store timestamps in my docs so that I query them. : : In trunk when I try to store a doc with a timestamp I get a sever error, is : there a different way I should store this data or is this a bug? are you sure your schema has that field defined as a (Trie)DateField ? Based on the error it looks like it's configured as a TrieDoubleField ... : 75004824785129473] Error adding field 'at_d'='2011-05-30T01:05:18Z' ... : org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) : Caused by: java.lang.NumberFormatException: For input string: : 2011-05-30T01:05:18Z : at : sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1222) : at java.lang.Double.parseDouble(Double.java:510) : at org.apache.solr.schema.TrieField.createField(TrieField.java:509) -Hoss
Re: saving timestamps in trunk broken?
I haven't modified my schema in the older solr or trunk solr,is it required to modify my schema to support timestamps? On Fri, Jul 22, 2011 at 4:45 PM, Chris Hostetter hossman_luc...@fucit.orgwrote: : In Solr 1.3.1 I am able to store timestamps in my docs so that I query them. : : In trunk when I try to store a doc with a timestamp I get a sever error, is : there a different way I should store this data or is this a bug? are you sure your schema has that field defined as a (Trie)DateField ? Based on the error it looks like it's configured as a TrieDoubleField ... : 75004824785129473] Error adding field 'at_d'='2011-05-30T01:05:18Z' ... : org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) : Caused by: java.lang.NumberFormatException: For input string: : 2011-05-30T01:05:18Z : at : sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1222) : at java.lang.Double.parseDouble(Double.java:510) : at org.apache.solr.schema.TrieField.createField(TrieField.java:509) -Hoss -- - sent from my mobile 6176064373
Spellcheck compounded words
How do I get spellchecker to suggest compounded words? Like. q=sail booat and suggestion/collate is sailboat and sail boat -- View this message in context: http://lucene.472066.n3.nabble.com/Spellcheck-compounded-words-tp3192748p3192748.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: saving timestamps in trunk broken?
This is the document I am posting: ?xml version=1.0 encoding=UTF-8?adddocfield name=idPost 75004824785129473/fieldfield name=typePost/fieldfield name=at_d2011-05-30T01:05:18Z/fieldfield name=state_sNew York/fieldfield name=country_sUnited States/fieldfield name=data_texthello world!/field/add In my schema.xml file I have these date fields, do I need more? fieldType name=date class=solr.TrieDateField omitNorms=true precisionStep=0 positionIncrementGap=0/ fieldType name=tdate class=solr.TrieDateField omitNorms=true precisionStep=6 positionIncrementGap=0/ fieldType name=pdate class=solr.DateField sortMissingLast=true omitNorms=true/ On Fri, Jul 22, 2011 at 5:00 PM, Jason Toy jason...@gmail.com wrote: I haven't modified my schema in the older solr or trunk solr,is it required to modify my schema to support timestamps? On Fri, Jul 22, 2011 at 4:45 PM, Chris Hostetter hossman_luc...@fucit.org wrote: : In Solr 1.3.1 I am able to store timestamps in my docs so that I query them. : : In trunk when I try to store a doc with a timestamp I get a sever error, is : there a different way I should store this data or is this a bug? are you sure your schema has that field defined as a (Trie)DateField ? Based on the error it looks like it's configured as a TrieDoubleField ... : 75004824785129473] Error adding field 'at_d'='2011-05-30T01:05:18Z' ... : org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) : Caused by: java.lang.NumberFormatException: For input string: : 2011-05-30T01:05:18Z : at : sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1222) : at java.lang.Double.parseDouble(Double.java:510) : at org.apache.solr.schema.TrieField.createField(TrieField.java:509) -Hoss -- - sent from my mobile 6176064373 -- - sent from my mobile 6176064373
Re: saving timestamps in trunk broken?
Hi Chris, you were correct, the filed was getting set as a double. Thanks for the help. On Fri, Jul 22, 2011 at 7:03 PM, Jason Toy jason...@gmail.com wrote: This is the document I am posting: ?xml version=1.0 encoding=UTF-8?adddocfield name=idPost 75004824785129473/fieldfield name=typePost/fieldfield name=at_d2011-05-30T01:05:18Z/fieldfield name=state_sNew York/fieldfield name=country_sUnited States/fieldfield name=data_texthello world!/field/add In my schema.xml file I have these date fields, do I need more? fieldType name=date class=solr.TrieDateField omitNorms=true precisionStep=0 positionIncrementGap=0/ fieldType name=tdate class=solr.TrieDateField omitNorms=true precisionStep=6 positionIncrementGap=0/ fieldType name=pdate class=solr.DateField sortMissingLast=true omitNorms=true/ On Fri, Jul 22, 2011 at 5:00 PM, Jason Toy jason...@gmail.com wrote: I haven't modified my schema in the older solr or trunk solr,is it required to modify my schema to support timestamps? On Fri, Jul 22, 2011 at 4:45 PM, Chris Hostetter hossman_luc...@fucit.org wrote: : In Solr 1.3.1 I am able to store timestamps in my docs so that I query them. : : In trunk when I try to store a doc with a timestamp I get a sever error, is : there a different way I should store this data or is this a bug? are you sure your schema has that field defined as a (Trie)DateField ? Based on the error it looks like it's configured as a TrieDoubleField ... : 75004824785129473] Error adding field 'at_d'='2011-05-30T01:05:18Z' ... : org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) : Caused by: java.lang.NumberFormatException: For input string: : 2011-05-30T01:05:18Z : at : sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1222) : at java.lang.Double.parseDouble(Double.java:510) : at org.apache.solr.schema.TrieField.createField(TrieField.java:509) -Hoss -- - sent from my mobile 6176064373 -- - sent from my mobile 6176064373 -- - sent from my mobile 6176064373
Problem in execution of error call in ajax request if solr server is not running
*$.ajax({
url:
solrURL+/solr/db/select/?qt=dismaxwt=jsonstart=+start+rows=+end+q=+query+json.wrf=?,
async:false,
dataType: 'json',
success: function(){
getSolrResponse(sort,order,itemPerPage,showPage,query,solrURL);
}
,error:function(){
alert(there is an error);
}
})*
I have this ajax request, but even if solr url is not correct, than also
error code is not executing, why so. please tell me the solution. i have
tried with both async:false and async:true. in both cases error not called
if server is shutdown
-
Thanks Regards
Romi
--
View this message in context:
http://lucene.472066.n3.nabble.com/Problem-in-execution-of-error-call-in-ajax-request-if-solr-server-is-not-running-tp3192962p3192962.html
Sent from the Solr - User mailing list archive at Nabble.com.
Error with custom search component which adds filter
I have a custom search component which does the following in process
SolrQueryRequest req = rb.req;
SolrParams params = req.getParams();
QueryWrapperFilter qwf = new QueryWrapperFilter(rb.getQuery());
Filter filter = new TestFilter(qwf);
ConstantScoreQuery fq = new ConstantScoreQuery(filter);
rb.setQuery(fq);
this essentially filters the result set using TestFilter which does
this in getDocIdSet
return new FilteredDocSet(startingFilter.getDocIdSet(readerCtx));
where FilteredDocSet always returns true from match. When a query is
executed that returns 0 results an IllegalArgumentException is thrown
from org.apache.lucene.search.FilteredDocIdSetIterator constructor
to fix this in FilteredDocSet I added the following:
@Override
public DocIdSetIterator iterator() throws IOException {
try {
return super.iterator();
} catch(IllegalArgumentException e){
return null;
}
}
What is the correct way to deal with this? Am I getting this error
because I'm misusing something?
