Re: "query" function query; what's it for?
On Sun, Nov 22, 2009 at 11:06 PM, David Smiley @MITRE.org wrote: > It's not clear to me what purpose the "query" function query solves. I've > read the description: > http://wiki.apache.org/solr/FunctionQuery#query but it doesn't really > explain the point of it. I'm sure it has to do with subtleties in how > scoring is done. Can someone please present a use-case? See the "Pure Nested Query" section here: http://www.lucidimagination.com/blog/2009/03/31/nested-queries-in-solr/ -Yonik http://www.lucidimagination.com
"query" function query; what's it for?
It's not clear to me what purpose the "query" function query solves. I've read the description: http://wiki.apache.org/solr/FunctionQuery#query but it doesn't really explain the point of it. I'm sure it has to do with subtleties in how scoring is done. Can someone please present a use-case? ~ David -- View this message in context: http://old.nabble.com/%22query%22-function-query--what%27s-it-for--tp26472546p26472546.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: access denied to solr home lib dir
Maybe ensuring that the full parent path (all parent directories) have "rx" permissions? -Yonik http://www.lucidimagination.com On Sun, Nov 22, 2009 at 2:59 PM, Charles Moad wrote: > I have been trying to get a new solr install setup on Ubuntu 9.10 > using tomcat6. I have tried the solr 1.4 release and the latest svn > for good measure. No matter what, I am running into the following > permission error. I removed all the lib includes from solrconfig.xml. > I have created the "/opt/solr/steve/lib" directory and all permissions > are good. This directory is optional, but I just cannot get past > this. I've installed solr 1.3 many times without running into this on > redhat boxes. > > Thanks, > Charlie > > Nov 22, 2009 2:48:53 PM org.apache.catalina.core.StandardContext filterStart > SEVERE: Exception starting filter SolrRequestFilter > org.apache.solr.common.SolrException: > java.security.AccessControlException: access denied > (java.io.FilePermission /opt/solr/steve/./lib read) > at > org.apache.solr.servlet.SolrDispatchFilter.(SolrDispatchFilter.java:68) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at java.lang.Class.newInstance0(Class.java:355) > at java.lang.Class.newInstance(Class.java:308) > at > org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:255) > at > org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:397) > at > org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:108) > at > org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3800) > at > org.apache.catalina.core.StandardContext.start(StandardContext.java:4450) > at > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791) > at > org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:123) > at > org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) > at java.security.AccessController.doPrivileged(Native Method) > at > org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:769) > at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:526) > at > org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:630) > at > org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:556) > at > org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:491) > at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1206) > at > org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:314) > at > org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119) > at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053) > at org.apache.catalina.core.StandardHost.start(StandardHost.java:722) > at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) > at > org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443) > at > org.apache.catalina.core.StandardService.start(StandardService.java:516) > at > org.apache.catalina.core.StandardServer.start(StandardServer.java:710) > at org.apache.catalina.startup.Catalina.start(Catalina.java:583) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:177) > Caused by: java.security.AccessControlException: access denied > (java.io.FilePermission /opt/solr/steve/./lib read) > at > java.security.AccessControlContext.checkPermission(AccessControlContext.java:323) > at > java.security.AccessController.checkPermission(AccessController.java:546) > at java.lang.SecurityManager.checkPermission(SecurityManager.java:532) > at java.lang.SecurityManager.checkRead(SecurityManager.java:871) > at java.io.File.canRe
Re: Solr 1.3 query and index perf tank during optimize
Oops, you're right, term listings and counts for deleted docs are adjusted during merges. I had the impression that optimize had some special powers here that merge does not. Thank you for bringing expungeDeletes to my attention. On Sat, Nov 21, 2009 at 7:46 AM, Yonik Seeley wrote: > On Sat, Nov 21, 2009 at 12:33 AM, Lance Norskog wrote: >> And, terms whose documents have been deleted are not purged. So, you >> can merge all you like and the index will not shrink back completely. > > Under what conditions? Certainly not all, since I just tried a simple > test and a merge removed the terms that were no longer in any > documents just fine. > >> This is important because the orphan terms affect relevance >> calculations. > > Marking a document as deleted don't affect any term statistics (which > idf uses) until the document is actually removed (which can happen via > a merge, optimize, or expungeDeletes). That's a lucene limitation > unrelated to how many of a terms documents have been deleted. But > perhaps I don't understand how you're using the term "orphan terms". > > -Yonik > http://www.lucidimagination.com > -- Lance Norskog goks...@gmail.com
Re: Weighting of synonyms
you can use payload and override defaultsimilrity for the wordnet synonmys check out http://www.lucidimagination.com/search/document/2d69478294303ab4/synonym_payload_boosting#39425a62e645c200 On Sun, Nov 22, 2009 at 22:03, L wrote: > Howdy, > > I recently rolled a custom WordNet synonym filter that pulls synonyms from > WordNet during indexing. All that is nice and dandy; however, it causes > problems in the sorting. Sometimes, the top match will come from a synonym > rather than the original word. > > An example in our system is a search on the word "party". The first match > was a doc with the word "company" which, in WordNet, has "party" as a > synonym. > > So the question is: is there a way to weight original word matches more > than synonym matches? In other words, I want all docs with "party" in the > text to come up first, then docs whose terms have "party" as a synonym. > > I'm hoping there is a way ;) It would seem this would be a common enough > issue. However, from my looking at the source code I fear the worst as I > don't see that you can do this unless it would make sense to utilize the > "type" field of Token. > > Thanks, > > Lajos > > > -- Regards _ David Ginzburg Developer, Digital Trowel 1 Hayarden St., Airport City [POB 169, NATBAG] Lod, 70151, Israel http://www.digitaltrowel.com/ Office: +972 73 240 522 Mobile: +972 50 496 0595 CHECK OUT OUR NEW TEXT MINING BLOG: http://mineyourbusiness.wordpress.com/
Weighting of synonyms
Howdy, I recently rolled a custom WordNet synonym filter that pulls synonyms from WordNet during indexing. All that is nice and dandy; however, it causes problems in the sorting. Sometimes, the top match will come from a synonym rather than the original word. An example in our system is a search on the word "party". The first match was a doc with the word "company" which, in WordNet, has "party" as a synonym. So the question is: is there a way to weight original word matches more than synonym matches? In other words, I want all docs with "party" in the text to come up first, then docs whose terms have "party" as a synonym. I'm hoping there is a way ;) It would seem this would be a common enough issue. However, from my looking at the source code I fear the worst as I don't see that you can do this unless it would make sense to utilize the "type" field of Token. Thanks, Lajos
access denied to solr home lib dir
I have been trying to get a new solr install setup on Ubuntu 9.10 using tomcat6. I have tried the solr 1.4 release and the latest svn for good measure. No matter what, I am running into the following permission error. I removed all the lib includes from solrconfig.xml. I have created the "/opt/solr/steve/lib" directory and all permissions are good. This directory is optional, but I just cannot get past this. I've installed solr 1.3 many times without running into this on redhat boxes. Thanks, Charlie Nov 22, 2009 2:48:53 PM org.apache.catalina.core.StandardContext filterStart SEVERE: Exception starting filter SolrRequestFilter org.apache.solr.common.SolrException: java.security.AccessControlException: access denied (java.io.FilePermission /opt/solr/steve/./lib read) at org.apache.solr.servlet.SolrDispatchFilter.(SolrDispatchFilter.java:68) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at java.lang.Class.newInstance0(Class.java:355) at java.lang.Class.newInstance(Class.java:308) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:255) at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:397) at org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:108) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3800) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4450) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:123) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:769) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:526) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:630) at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:556) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:491) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1206) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:314) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053) at org.apache.catalina.core.StandardHost.start(StandardHost.java:722) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443) at org.apache.catalina.core.StandardService.start(StandardService.java:516) at org.apache.catalina.core.StandardServer.start(StandardServer.java:710) at org.apache.catalina.startup.Catalina.start(Catalina.java:583) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:177) Caused by: java.security.AccessControlException: access denied (java.io.FilePermission /opt/solr/steve/./lib read) at java.security.AccessControlContext.checkPermission(AccessControlContext.java:323) at java.security.AccessController.checkPermission(AccessController.java:546) at java.lang.SecurityManager.checkPermission(SecurityManager.java:532) at java.lang.SecurityManager.checkRead(SecurityManager.java:871) at java.io.File.canRead(File.java:689) at org.apache.solr.core.SolrResourceLoader.replaceClassLoader(SolrResourceLoader.java:157) at org.apache.solr.core.SolrResourceLoader.addToClassLoader(SolrResourceLoader.java:128) at org.apache.solr.core.SolrResourceLoader.(SolrResourceLoader.java:97) at org.apache.solr.
Re: Question about lat/long data type in localsolr
Sorry, I forgot to mention that I already did double -> tdouble replacement. Yesterday, I downloaded localsolr https://locallucene.svn. sourceforge.net/svnroot/locallucene/trunk/localsolr and use lucene-spatial-2.9.0-dev.jar. The indexing is succcessful, although currently I am still figuring out why the search returns 0 result or throw an exception. Thanks. On Sat, Nov 21, 2009 at 1:43 PM, Ian Ibbotson wrote: > Heya... > > I think you need to use the newer types in your schema.xml, IE > > > > > > as doubles are no longer index-compatible (AFAIK) > > To use the above, make sure you have the tdouble types declared with > > precisionStep="8" omitNorms="true" positionIncrementGap="0"/> > > in your section. > > HTH > > Ian. > > > 2009/11/21 Bertie Shen : > > Hey everyone, > > > > I used localsolr and locallucene to do local search. But I could not > make > > longitude and latitude successfully indexed. During DataImport process, > > there is an exception. Do you have some ideas about it? > > > > I copy solrconfig.xml and schema.xml from your > > http://www.gissearch.com/localsolr. Only change I made is to replace > names > > lat and lng by latitude and longtitude respectively, which are field name > in > > my index. Should "str" in lat should be > replaced > > by "double" according to the following exception? > > > > Thanks. > > > > Solr log about Exception. > > > > > >java.lang.ClassCastException: java.lang.Double cannot be cast > > to java.lan\ > > g.String > > > > com.pjaol.search.solr.update.LocalUpdaterProcessor > > processAdd > > 136 > > > > > > org.apache.solr.handler.dataimport.SolrWriter > > upload > > 75 > > > > > > > org.apache.solr.handler.dataimport.DataImportHandler$1 > > upload > > 292 > > > > > > org.apache.solr.handler.dataimport.DocBuilder > > buildDocument > > 392 > > > > > > org.apache.solr.handler.dataimport.DocBuilder > > doFullDump > > 242 > > > > > > org.apache.solr.handler.dataimport.DocBuilder > > execute > > 180 > > > > > > org.apache.solr.handler.dataimport.DataImporter > > doFullImport > > 331 > > > > > > org.apache.solr.handler.dataimport.DataImporter > > runCmd > > 389 > > > > > > org.apache.solr.handler.dataimport.DataImporter$1 > > run > > 370 > > > > > > > > > > How do I set up local indexing > > > > Here is what I have done to set up local indexing. > > 1) Download localsolr. I download it from > > http://developer.k-int.com/m2snapshots/localsolr/localsolr/1.5/ and put > jar > > file (in my case, localsolr-1.5.jar) in your application's WEB_INF/lib > > directory of application server. > > > > 2) Download locallucene. I download it from > > http://sourceforge.net/projects/locallucene/ and put jar file (in my > case, > > locallucene.jar in locallucene_r2.0/dist/ diectory) in your application's > > WEB_INF/lib directory of application server. I also need to copy > > gt2-referencing-2.3.1.jar, geoapi-nogenerics-2.1-M2.jar, and > jsr108-0.01.jar > > under locallucene_r2.0/lib/ directory to WEB_INF/lib. Do not copy > > lucene-spatial-2.9.1.jar under Lucene codebase. The namespace has been > > changed from com.pjaol.blah.blah.blah to org.apache.blah blah. > > > > 3) Update your solrconfig.xml and schema.xml. I copy it from > > http://www.gissearch.com/localsolr. > > >
Re: Index time boosts, payloads, and long query strings
I still think they are apples and oranges. If you boost *all* titles, you're effectively boosting none of them. Index time boosting expresses "this document's title is more important than other document titles." What I think you're after is "titles are more important than other parts of the document. For this latter, you're talking query-time boosting. Boosting only really makes sense if there are multiple clauses, something like title:important OR body:unimportant. If this is true, speed is irrelevant, you need correct behavior. Not that I think you'd notice either way. Modern computers can do a LOT of FLOPS/sec. Here's an experiment: time some queries (but beware of timing the very first ones, see the Wiki) with boosts and without boosts. I doubt you'll see enough difference to matter (but please do report back if you do, it'll further my education ). But, depending on your index structure, you may get this anyway. Generally, matches on shorter fields weigh more in the score calculations than on longer fields. If you have fields like title and body and you are querying on title:term OR body:term, documents with term in the title will tend toward higher scores. But before putting too much effort into this, do you have any evidence that the default behavior is unsatisfactory? Because unless and until you do, I think this is a distraction ... Best Erick On Sun, Nov 22, 2009 at 8:37 AM, Girish Redekar wrote: > Hi Erick - > > Maybe I mis-wrote. > > My question is: would "title:any_query^4.0" be faster/slower than applying > index time boost to the field title. Basically, if I take *every* user > query > and search for it in title with boost (say, 4.0) - is it different than > saying field title has boost 4.0? > > Cheers, > Girish Redekar > http://girishredekar.net > > > On Sun, Nov 22, 2009 at 2:02 AM, Erick Erickson >wrote: > > > I'll take a whack at index .vs. query boosting. They are expressing very > > different concepts. Let's claim we're interested in boosting the title > > field > > > > Index time boosting is expressing "this document's title is X more > > important > > > > than a normal document title". It doesn't matter *what* the title is, > > any query that matches on anything in this document's title will give > this > > document a boost. I might use this to give preferential treatment to all > > encyclopedia entries or something. > > > > Query time boosting, like "title:solr^4.0" expresses "Any document with > > solr > > in > > it's title is more important than documents without solr in the title". > > This > > really > > only makes sense if you have other clauses that might cause a document > > *without* > > solr the title to match.. > > > > Since they are doing different things, efficiency isn't really relevant. > > > > HTH > > Erick > > > > > > On Sat, Nov 21, 2009 at 2:13 AM, Girish Redekar > > wrote: > > > > > Hi , > > > > > > I'm relatively new to Solr/Lucene, and am using Solr (and not lucene > > > directly) primarily because I can use it without writing java code > (rest > > of > > > my project is python coded). > > > > > > My application has the following requirements: > > > (a) ability to search over multiple fields, each with different weight > > > (b) If possible, I'd like to have the ability to add extra/diminished > > > weights to particular tokens within a field > > > (c) My query strings have large lengths (50-100 words) > > > (d) My index is 500K+ documents > > > > > > 1) The way to (a) is field boosting (right?). My question is: Is all > > field > > > boosting done at query time? Even if I give index time boosts to > fields? > > Is > > > there a performance advantage in boosting fields at index time vs at > > using > > > something like fieldname:querystring^boost. > > > 2) From what I've read, it seems that I can do (b) using payloads. > > However, > > > as this link ( > > > > > > > > > http://www.lucidimagination.com/blog/2009/08/05/getting-started-with-payloads/ > > > ) > > > suggests, I will have to write a payload aware Query Parser. Wanted to > > > confirm if this is indeed the case - or is there a out-of-box way to > > > implement payloads (am using Solr1.4) > > > 3) For my project, the user fills multiple text boxes (for each query). > I > > > combine these into a single query (with different treatment for > contents > > of > > > each text box). Consequently, my query looks something like > (fieldname1: > > > queryterm1 queryterm2^2.0 queryterm3^3.0 +queryterm4)^1.0 Are there > any > > > guidelines for improving performance of such a system (sorry, this bit > is > > > vague) > > > > > > Any help with this will be great ! > > > > > > Girish Redekar > > > http://girishredekar.net > > > > > >
Re: Index time boosts, payloads, and long query strings
Hi Erick - Maybe I mis-wrote. My question is: would "title:any_query^4.0" be faster/slower than applying index time boost to the field title. Basically, if I take *every* user query and search for it in title with boost (say, 4.0) - is it different than saying field title has boost 4.0? Cheers, Girish Redekar http://girishredekar.net On Sun, Nov 22, 2009 at 2:02 AM, Erick Erickson wrote: > I'll take a whack at index .vs. query boosting. They are expressing very > different concepts. Let's claim we're interested in boosting the title > field > > Index time boosting is expressing "this document's title is X more > important > > than a normal document title". It doesn't matter *what* the title is, > any query that matches on anything in this document's title will give this > document a boost. I might use this to give preferential treatment to all > encyclopedia entries or something. > > Query time boosting, like "title:solr^4.0" expresses "Any document with > solr > in > it's title is more important than documents without solr in the title". > This > really > only makes sense if you have other clauses that might cause a document > *without* > solr the title to match.. > > Since they are doing different things, efficiency isn't really relevant. > > HTH > Erick > > > On Sat, Nov 21, 2009 at 2:13 AM, Girish Redekar > wrote: > > > Hi , > > > > I'm relatively new to Solr/Lucene, and am using Solr (and not lucene > > directly) primarily because I can use it without writing java code (rest > of > > my project is python coded). > > > > My application has the following requirements: > > (a) ability to search over multiple fields, each with different weight > > (b) If possible, I'd like to have the ability to add extra/diminished > > weights to particular tokens within a field > > (c) My query strings have large lengths (50-100 words) > > (d) My index is 500K+ documents > > > > 1) The way to (a) is field boosting (right?). My question is: Is all > field > > boosting done at query time? Even if I give index time boosts to fields? > Is > > there a performance advantage in boosting fields at index time vs at > using > > something like fieldname:querystring^boost. > > 2) From what I've read, it seems that I can do (b) using payloads. > However, > > as this link ( > > > > > http://www.lucidimagination.com/blog/2009/08/05/getting-started-with-payloads/ > > ) > > suggests, I will have to write a payload aware Query Parser. Wanted to > > confirm if this is indeed the case - or is there a out-of-box way to > > implement payloads (am using Solr1.4) > > 3) For my project, the user fills multiple text boxes (for each query). I > > combine these into a single query (with different treatment for contents > of > > each text box). Consequently, my query looks something like (fieldname1: > > queryterm1 queryterm2^2.0 queryterm3^3.0 +queryterm4)^1.0 Are there any > > guidelines for improving performance of such a system (sorry, this bit is > > vague) > > > > Any help with this will be great ! > > > > Girish Redekar > > http://girishredekar.net > > >
