error in solr.add(docs);
error faced on line solrCore.add(docs); Exception in thread main java.lang.NoSuchMethodError: org.apache.solr.common.params.SolrParams.wrapDefaults(Lorg/apache/solr/common/params/SolrParams;Lorg/apache/solr/common/params/SolrParams;)Lorg/apache/solr/common/params/SolrParams; at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:272) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:249) at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105) at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:69) at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54) at solr.performance.Test.add_docs(Test.java:83) at solr.performance.Test.main(Test.java:103) -- Thanks Regards Sachin Aggarwal 7760502772
Re: error in solr.add(docs);
i found it error was appearing because i included a a lib file solr-common-1.3.0.jar On Thu, Jul 12, 2012 at 1:34 PM, Sachin Aggarwal different.sac...@gmail.com wrote: error faced on line solrCore.add(docs); Exception in thread main java.lang.NoSuchMethodError: org.apache.solr.common.params.SolrParams.wrapDefaults(Lorg/apache/solr/common/params/SolrParams;Lorg/apache/solr/common/params/SolrParams;)Lorg/apache/solr/common/params/SolrParams; at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:272) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:249) at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105) at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:69) at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54) at solr.performance.Test.add_docs(Test.java:83) at solr.performance.Test.main(Test.java:103) -- Thanks Regards Sachin Aggarwal 7760502772 -- Thanks Regards Sachin Aggarwal 7760502772
Re: Problems with elevation component configuration
Hi, Thanks Chris, but actually, it turns out that query text from elevate.xml has to match the query (q=...). So in this case, elevation works only for http://localhost:8080/solr/elevate?q=brain, but not for http://localhost:8080/solr/elevate?q=indexingabstract:brain type of queries. This could be solved by using DisMax query parser (http://localhost:8080/solr/elevate?q=brainqf=indexingabstractdefType=dismax), but we have way more complicated queries which cannot be reduced just to q=searchterm... Can anything be done about that without modifying QueryElevationComponent class? Thanks, Igor -- View this message in context: http://lucene.472066.n3.nabble.com/Problems-with-elevation-component-configuration-tp3993204p3994595.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: DataImport using last_indexed_id or getting max(id) quickly
Hi Avenka,
you asked for a HowTo to add a field inverseID which allows to calculate
max(id) from its first term:
If you do not use solr you have to calculate 1 - id and store it in
an extra field inverseID.
If you fill solr with your own code, add a TrieLongField inverseID and fill
with the value -id.
If you only want to change schema.xml (and add some classes):
* You need a new FieldType inverseLongType and a Field inverseID of Type
inverseLongType
* You need a line copyField source=id dest=inverseID/
(see http://wiki.apache.org/solr/SchemaXml#Copy_Fields)
For inverseLongType I see two possibilities
a) use TextField and make your own filter to calculate 1 - id
b) extends TrieLongField to a new FieldType InverseTrieLongField with:
@Override
public String readableToIndexed(String val) {
return super.readableToIndexed(Long.toString( -Long.parseLong(val)));
}
@Override
public Fieldable createField(SchemaField field, String externalVal, float
boost) {
return super.createField(field,Long.toString( -Long.parseLong(val)), boost
);
}
@Override
public Object toObject(Fieldable f) {
Object result = super.toObject(f);
if(result instanceof Long){
return new Long( -((Long)result).longValue());
}
return result;
}
Beste regards
Karsten
View this message in context:
http://lucene.472066.n3.nabble.com/DataImport-using-last-indexed-id-or-getting-max-id-quickly-tp3993763p3994560.html
Original-Nachricht
Datum: Wed, 11 Jul 2012 20:59:10 -0700 (PDT)
Von: avenka ave...@gmail.com
An: solr-user@lucene.apache.org
Betreff: Re: DataImport using last_indexed_id or getting max(id) quickly
Thanks. Can you explain more the first TermsComponent option to obtain
max(id)? Do I have to modify schema.xml to add a new field? How exactly do
I
query for the lowest value of 1 - id?
--
View this message in context:
http://lucene.472066.n3.nabble.com/DataImport-using-last-indexed-id-or-getting-max-id-quickly-tp3993763p3994560.html
Sent from the Solr - User mailing list archive at Nabble.com.
NRT and multi-value facet - what is Solr's limit?
Hi, I understand that the cache for multi-value facet is multi-segment. So every time a document is updated the entire cache needs to be rebuilt. Is there any rule of thumb on the highest update rate NRT can handle before this cache-rebuild-on-each-commit becomes too expensive? I know it depends, but I'm just looking for order-of-magnitude estimates. Are we talking about 10 updates/s? 100? 1,000? Thanks
Re: shard connection timeout
Hi, Jason, That's a huge heap. Which Directory implementation are you using? It might make more sense to drastically reduce that heap and let the OS buffer the index at a lower level. Michael Della Bitta Appinions, Inc. -- Where Influence Isn’t a Game. http://www.appinions.com On Wed, Jul 11, 2012 at 10:05 PM, Jason hialo...@gmail.com wrote: Hi Erick Our physical memory is 128g and jvm option is -Xms110g -Xmx110g -XX:PermSize=512m -XX:MaxPermSize=512m This is very huge memory allocation. But we are servicing patent search and that is accepted very complex queries used memory. -- View this message in context: http://lucene.472066.n3.nabble.com/shard-connection-timeout-tp3994301p3994556.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: NRT and multi-value facet - what is Solr's limit?
Hi Andy, as long as the cache for facetting is not per segment there is no NRT together with facetting. This is what Jason told you in http://lucene.472066.n3.nabble.com/Nrt-and-caching-td3993612.html and I am agree. Possible you could use multicore. Beste regards Karsten Original-Nachricht Datum: Thu, 12 Jul 2012 03:18:47 -0700 (PDT) Von: Andy angelf...@yahoo.com An: solr-user@lucene.apache.org solr-user@lucene.apache.org Betreff: NRT and multi-value facet - what is Solr\'s limit? Hi, I understand that the cache for multi-value facet is multi-segment. So every time a document is updated the entire cache needs to be rebuilt. Is there any rule of thumb on the highest update rate NRT can handle before this cache-rebuild-on-each-commit becomes too expensive? I know it depends, but I'm just looking for order-of-magnitude estimates. Are we talking about 10 updates/s? 100? 1,000? Thanks
Re: query syntax to find ??? chars
don't know why but doesn't work. :( Best Regards Alexander Aristov On 11 July 2012 23:54, Yury Kats yuryk...@yahoo.com wrote: On 7/11/2012 2:55 PM, Alexander Aristov wrote: content:?? doesn't work :) I would try escaping them: content:\?\?\?\?\?\?
multiValued false-true
I have an indexed, not stored, not multiValued field in the schema. If I change this field to be multiValued, would I need to re-index everything, or would all existing documents (that were indexed while the field was not multiValued) still be queryable? Thanks, Yury
Re: query syntax to find ??? chars
Perhaps they're being displayed as question marks, but the actual character is different? Michael Della Bitta Appinions, Inc. -- Where Influence Isn’t a Game. http://www.appinions.com On Thu, Jul 12, 2012 at 9:30 AM, Alexander Aristov alexander.aris...@gmail.com wrote: don't know why but doesn't work. :( Best Regards Alexander Aristov On 11 July 2012 23:54, Yury Kats yuryk...@yahoo.com wrote: On 7/11/2012 2:55 PM, Alexander Aristov wrote: content:?? doesn't work :) I would try escaping them: content:\?\?\?\?\?\?
Re: Nrt and caching
Hi Andy, Multi-value faceting is a special case of taxonomy. So it is covered by the org.apache.lucene.facet package (lucene/facet). This is not per segment but works without per IndexSearcher cache. So imho the taxonomy faceting will work with NRT. Because of the new TermsEnum#ord() Method the class UnInvertedField already lost half of its code-lines. UnInvertedField would work per segment, if the ordinal position for a term would not change in a commit. Which is the basic idea of the taxonomy-solution. So I am quite sure that Solr will adopt this approach any time. I do not now about soon. Best regards Karsten in context: http://lucene.472066.n3.nabble.com/Nrt-and-caching-tp3993612p3993700.html Original-Nachricht Datum: Sat, 7 Jul 2012 17:32:52 -0700 (PDT) Von: Andy angelf...@yahoo.com An: solr-user@lucene.apache.org solr-user@lucene.apache.org Betreff: Re: Nrt and caching Jason, If I just use stock Solr 4.0 without modifying the source code, does that mean multi-value faceting will be very slow when I'm constantly inserting/updating documents? Which open source library are you referring to? Will Solr adopt this per-segment approach any time soon? Thanks
Re: Trunk error in Tomcat
it works, with a few changes :) I think we don't need a new issue in jura. Solr 4.0 is no longer Solr 4.0 since late february. There were some changes in solrconfig.xml in this time. I migrate my solr 4.0 trunk-config, which works till late february in a new config from 4.0 alpha. A couple of changes which i noticed: - abortOnConfigurationError:true is gone - luceneMatchVersion was changed to LUCENE_50 - a couple of new jars included for velocity and lang - new directory Factory = solr.directoryFactory:solr.NRTCachingDirectoryFactory - indexDefaults replaced by indexConfig - updateLog added - replication Handler for SoldCloud added - Names for handlers were changed like /select for search - new Handler added requestHandler name=/get class=solr.RealTimeGetHandler and so on... This AdminHandler-Exception is still there, when i use the clusteringComponent, see here: SCHWERWIEGEND: null:org.apache.solr.common.SolrException: Error loading class 'solr.clustering.ClusteringComponent' But if i comment it out, Solr starts without errors. The paths to the clustering jar in ../contrib/clustering/lib/ is correct and the needed jars are there, eventually we need new jar-files? Best regards Vadim 2012/7/5 Stefan Matheis matheis.ste...@googlemail.com: Great, thanks Vadim On Thursday, July 5, 2012 at 9:34 AM, Vadim Kisselmann wrote: Hi Stefan, ok, i would test the latest version from trunk with tomcat in next days and open an new issue:) regards Vadim 2012/7/3 Stefan Matheis matheis.ste...@googlemail.com (mailto:matheis.ste...@googlemail.com): On Tuesday, July 3, 2012 at 8:10 PM, Vadim Kisselmann wrote: sorry, i overlooked your latest comment with the new issue in SOLR-3238 ;) Should i open an new issue? NP Vadim, yes a new Issue would help .. all available Information too :)
Re: Updating documents
Vinicius: No, fetching the document from the index, changing selected values and re-indexing probably won't work at all. The problem is that you only get _stored_ values back from Solr. So unless you've specified 'stored=true ' for all your fields, you can't use the doc fetched from Solr to update a field. The partial documents update that Jonatan references also requires that all the fields be stored. You're best bet is to go back to your system-of-record for the data and re-index the whole document. Best Erick On Wed, Jul 11, 2012 at 11:30 AM, Jonatan Fournier jonatan.fourn...@gmail.com wrote: On Wed, Jul 11, 2012 at 10:57 AM, Vinicius Carvalho viniciusccarva...@gmail.com wrote: Hi there. I was checking the faq and found that solr does not support field updates right. So I assume that in order to update a document, one should first retrieve it by its Id and then change the required field and update the doc again. But then I wonder about fields that are indexed and not stored, since the new document that is sent to the index does not have the values, would this mean we will loose them? BTW any chances we see field level updates on 4.0 like elastic search has? I'm actually also looking a this new feature in 4.0-ALPHA: http://solr.pl/en/2012/07/09/solr-4-0-partial-documents-update/ I was wondering where the new xml tags where documented to support these set, add to multi-value etc. -- jonatan Regards -- The intuitive mind is a sacred gift and the rational mind is a faithful servant. We have created a society that honors the servant and has forgotten the gift.
Re: Sort by date field = outofmemory?
Bruno: You can also reduce your memory requirements by storing fewer unique values. All the _unique_ values for a field in the index are read in for sorting. People often store timestamps in milliseconds, which essentially means that every document has a unique value. Storing your timestamps in the coarsest granularity that suits your use-case is always a good idea, see the date math: http://lucene.apache.org/solr/api-4_0_0-ALPHA/org/apache/solr/util/DateMathParser.html Best Erick On Wed, Jul 11, 2012 at 12:44 PM, Yury Kats yuryk...@yahoo.com wrote: This solves the problem by allocating memory up front, instead of at some point later when JVM needs it. At that later point in time there may not be enough free memory left on the system to allocate. On 7/11/2012 11:04 AM, Michael Della Bitta wrote: There is a school of thought that suggests you should always set Xms and Xmx to the same thing if you expect your heap to hit Xms. This results in your process only needing to allocate the memory once, rather in a series of little allocations as the heap expands. I can't explain how this fixed your problem, but just a datapoint that might suggest that doing what you did is not such a bad thing. Michael Della Bitta Appinions, Inc. -- Where Influence Isn’t a Game. http://www.appinions.com On Wed, Jul 11, 2012 at 4:05 AM, Bruno Mannina bmann...@free.fr wrote: Hi, some news this morning... I added -Xms1024m option and now it works?! no outofmemory ?! java -jar -Xms1024m -Xmx2048m start.jar Le 11/07/2012 09:55, Bruno Mannina a écrit : Hi Yury, Thanks for your anwer. ok for to increase memory but I have a problem with that, I have 8Go on my computer but the JVM accepts only 2Go max with the option -Xmx is it normal? Thanks, Bruno Le 11/07/2012 03:42, Yury Kats a écrit : Sorting is a memory-intensive operation indeed. Not sure what you are asking, but it may very well be that your only option is to give JVM more memory. On 7/10/2012 8:25 AM, Bruno Mannina wrote: Dear Solr Users, Each time I try to do a request with sort=pubdate+desc I get: GRAVE: java.lang.OutOfMemoryError: Java heap space I use Solr3.6, I have around 80M docs and my request gets around 160 results. Actually for my test, i use jetty java -jar -Xmx2g start.jar PS: If I write 3g i get an error, I have 8go Ram Thanks a lot for your help, Bruno
Re: Solr facet multiple constraint
Sure, you can have as complex an FQ clause as you need. Best Erick On Thu, Jul 12, 2012 at 1:48 AM, davidbougearel david.bougea...@smile-benelux.com wrote: Ok this is the id but in fact (sorry about this) my wish is the reverse, i want to get just the facet for which a have the right so i want to put fq=user:10 in order to get only facet with user:10. In my fq i can have something like user:10 AND user:3 because it's auto generated by rights of my application. Have you any idea to solve this ? Really thanks to help me, we are waiting a solution for this because instead of this we should make a lot of queries to get the right result. Best regards, David. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-facet-multiple-constraint-tp3992974p3994574.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: query syntax to find ??? chars
On 12 July 2012 19:28, Michael Della Bitta michael.della.bi...@appinions.com wrote: Perhaps they're being displayed as question marks, but the actual character is different? This is very likely to be the case, as messing up the encoding can leave one with entirely unexpected characters. The '?' probably corresponds to a non-displayable character. One way to find the actual characters is to examine the content in a capable Unicode editor. In my experience, yudit ( http://yudit.org ) has the best Unicode support, and will show you the actual hex code-point even for non-displayable characters. The bad news is that there is probably now way to recover the actual encoding. Plus, it might be difficult to have a generic way of identifying such documents. Regards, Gora
Re: Updating documents
Erick, On Thu, Jul 12, 2012 at 11:05 AM, Erick Erickson erickerick...@gmail.com wrote: Vinicius: No, fetching the document from the index, changing selected values and re-indexing probably won't work at all. The problem is that you only get _stored_ values back from Solr. So unless you've specified 'stored=true ' for all your fields, you can't use the doc fetched from Solr to update a field. The partial documents update that Jonatan references also requires that all the fields be stored. If my only fields with stored=false are copyField (e.g. I don't need their content to rebuild the document), are they gonna be re-copied with the partial document update? -- jonatan You're best bet is to go back to your system-of-record for the data and re-index the whole document. Best Erick On Wed, Jul 11, 2012 at 11:30 AM, Jonatan Fournier jonatan.fourn...@gmail.com wrote: On Wed, Jul 11, 2012 at 10:57 AM, Vinicius Carvalho viniciusccarva...@gmail.com wrote: Hi there. I was checking the faq and found that solr does not support field updates right. So I assume that in order to update a document, one should first retrieve it by its Id and then change the required field and update the doc again. But then I wonder about fields that are indexed and not stored, since the new document that is sent to the index does not have the values, would this mean we will loose them? BTW any chances we see field level updates on 4.0 like elastic search has? I'm actually also looking a this new feature in 4.0-ALPHA: http://solr.pl/en/2012/07/09/solr-4-0-partial-documents-update/ I was wondering where the new xml tags where documented to support these set, add to multi-value etc. -- jonatan Regards -- The intuitive mind is a sacred gift and the rational mind is a faithful servant. We have created a society that honors the servant and has forgotten the gift.
Re: Updating documents
On Thu, Jul 12, 2012 at 12:38 PM, Jonatan Fournier jonatan.fourn...@gmail.com wrote: On Thu, Jul 12, 2012 at 11:05 AM, Erick Erickson The partial documents update that Jonatan references also requires that all the fields be stored. If my only fields with stored=false are copyField (e.g. I don't need their content to rebuild the document), are they gonna be re-copied with the partial document update? Correct - your setup should be fine. Only original source fields (non copyField targets) should have stored=true -Yonik http://lucidimagination.com
Re: multiValued false-true
: I have an indexed, not stored, not multiValued field in the schema. : : If I change this field to be multiValued, would I need to re-index : everything, or would all existing documents (that were indexed while : the field was not multiValued) still be queryable? usually a change like this is fine ... the existing indexed values will still be searchable. when a change like this may not work the work the way you'd expect is if you have some special plugin (like an update processor for example) that changes behavior based on wether a field is multivalued or not -- your existing indexed values will still be there and still be searchable, but the index you have may not be the same as the index you'd have if you rebuilt from scratch after making the schema change. -Hoss
Case-insensitive on facet prefix
Hello all. I have a field configured with the LowerCaseFilterFactory as the only analyzer (for botth indexing and searching). The problem is that facet.prefix doesn't work on that field as expected. For example: Indexed term: house -- LowerCaseFilterFactory applied facet.prefix=hou -- returns a house entry as expected facet.prefix=Hou -- no match I suppose the LowerCaseFilterFactory it's not been applied on this prefix term. So ... is this the expected behavior ? How can I perform a facet with a case-insensitive prefix ? Thanks in advance Nestor SeDiCI - http://sedici.unlp.edu.ar PrEBi - http://prebi.unlp.edu.ar Universidad Nacional de La Plata La Plata, Buenos Aires, Argentina
Re: Updating documents
Yonik,
On Thu, Jul 12, 2012 at 12:52 PM, Yonik Seeley
yo...@lucidimagination.com wrote:
On Thu, Jul 12, 2012 at 12:38 PM, Jonatan Fournier
jonatan.fourn...@gmail.com wrote:
On Thu, Jul 12, 2012 at 11:05 AM, Erick Erickson
The partial documents update that Jonatan references also requires
that all the fields be stored.
If my only fields with stored=false are copyField (e.g. I don't need
their content to rebuild the document), are they gonna be re-copied
with the partial document update?
Correct - your setup should be fine. Only original source fields (non
copyField targets) should have stored=true
Another question I had related to partial update...
$ ./post.sh foo.json
{responseHeader:{status:409,QTime:0},error:{msg:Document
not found for update. id=foo,code:409}}
Is there a flag for: if document does not exist, create it for me? The
thing is that I don't know in advance if the document already exist
(of course I could query first.. but I have millions of entry to
process, might exist, might be an update I don't know...)
My naive approach was to have in the same request two documents, one
with only set using the unique ID, and then in the second one all
the add (concerning multivalue field).
So it would do the following:
1. Document (with id) exist or not don't care, use the following set
command to update/create
2. 2nd pass, I know you exist (with above id), please add all those to
the multivalue fields (none of those fields are in the initial
updates)
My rationale is that if the document exists, reset some fields, and
then append the multivalue fields (those multivalue fields express
historical updates)
The reason I created 2 documents is that Solr doesn't seem happy if I
mix set and add in the same document :)
--
jonatan
-Yonik
http://lucidimagination.com
Solr custom XSLT with highlighting
I have created the following sample xslt which highlights the keyword in the
content field for anybody who wants to use it.
1) In your solrconfig.xml
requestHandler name=/select class=solr.SearchHandler
lst name=defaults
...
Add following params to this defaults section
str name=hlon/str
str name=flhighlighting,title,url/str
str name=wtxslt/str
str name=trcustom-solr.xsl/str
str name=hl.fltitle,content/str
2) Create the following custom-solr.xsl under /solr/example/solr/conf/xslt
---custom-solr.xsl (start)--[solr
3.0.6]--
?xml version='1.0' encoding='UTF-8'?
xsl:transform version=2.0
xmlns:xsl=http://www.w3.org/1999/XSL/Transform;
xmlns:xs=http://www.w3.org/2001/XMLSchema;
exclude-result-prefixes=xs
xsl:output method=xml version=1.0 encoding=UTF-8 indent =yes/
xsl:output method=html indent=yes version=4.0/
xsl:variable name=title select=concat('Total
(',response/result/@numFound,' documents) found')/
xsl:variable name=keyword select=response/lst/lst/str[@name='q']/
xsl:template match='/'
html
head
titlexsl:value-of select=$title//title
xsl:call-template name=css/
/head
body
xsl:value-of select=$title/ matching {xsl:value-of select=$keyword/}
xsl:apply-templates select=response/
/body
/html
/xsl:template
xsl:template match=response
xsl:for-each select=result/doc
xsl:variable name=cTitle select=arr[@name='title']/str/
xsl:variable name=cUrl select=str[@name='url']/
xsl:variable name=hContent
select=/response/lst[@name='highlighting']/lst[@name=$cUrl]/arr/str/
xsl:attribute name=hrefxsl:value-of
select=$cUrl//xsl:attribute
xsl:call-template name=unescapeEm
xsl:with-param name=val select=$cTitle/
/xsl:call-template
div class=note
label.../label
xsl:call-template name=unescapeEm
xsl:with-param name=val select=$hContent/
/xsl:call-template
/div
/xsl:for-each
/xsl:template
xsl:template name=unescapeEm
xsl:param name=val select=''/
xsl:variable name=preEm select=substring-before($val, 'lt;')/
xsl:choose
xsl:when test=$preEm or starts-with($val, 'lt;')
xsl:variable name=insideEm
select=substring-before($val,'lt;/')/
xsl:value-of select=$preEm/strongxsl:value-of
select=substring($insideEm, string-length($preEm)+5)//strong
xsl:variable name=leftover select=substring($val,
string-length($insideEm) + 6)/
xsl:if test=$leftover
xsl:call-template name=unescapeEm
xsl:with-param name=val select=$leftover/
/xsl:call-template
/xsl:if
/xsl:when
xsl:otherwise
xsl:value-of select=$val/
/xsl:otherwise
/xsl:choose
label.../label
/xsl:template
xsl:template match=*/
xsl:template name=css
/xsl:template
/xsl:transform
---custom-solr.xsl (end)-
--
View this message in context:
http://lucene.472066.n3.nabble.com/Solr-custom-XSLT-with-highlighting-tp3994704.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Problems with elevation component configuration
: Thanks Chris, but actually, it turns out that query text from elevate.xml : has to match the query (q=...). So in this case, elevation works only for : http://localhost:8080/solr/elevate?q=brain, but not for : http://localhost:8080/solr/elevate?q=indexingabstract:brain type of queries. right ... query elevation by default is based on the raw query string. : but we have way more complicated queries which cannot be reduced just to : q=searchterm... what would you want/exect QEC to do in that type of situation? how would it know what part of a complex query should/shouldn't be used for elevation? FWIW: ne thing you can do is configure a queryFieldType on in your QueryElevationComponent .. if specific, i will use the analyzer for that field type to process the raw query string before doing a lookup in the QEC data -- so for example: you could use it to lowercase the input, or strip out unwanted whitespace or punctuation. it night not help for really complicated queries, but it would let you easily deal with things like extra whitespace you want to ignore. I think it would also be fairly easy to make QEC support an elevate.q param similar to how there is a spellcheck.q param and a hl.q param to let the client specify an alternate, simplified, string for the feature to use -- feel free to open an issue for that and take a crack at a patch if you're interested. -Hoss
Re: Solr ids query parameter
I'm pretty sure a sharded query/response gets tagged in a special way that might be incompatible with multiple tiers like this. Had you considered a single tier that consisted of a load-balanced pool of 'aggregator' servers? Michael Della Bitta Appinions, Inc. -- Where Influence Isn’t a Game. http://www.appinions.com On Thu, Jul 12, 2012 at 5:36 PM, Peter Markey sudoma...@gmail.com wrote: Hello, I am trying to do a distributed search with solr and for some reason I get an internal server error. The set up is like this: I have 4 solr servers that index data (say daily each with 10 cores) and I use another bunch of solr instances (lets call one of them as L1aggregator) that does a distributed request to all the 40 cores of 4 solr servers. I also have another solr instance (lets call it L2aggregator) that would be collecting data from multiple L1aggregator solr instances. Its like a multi-level structure. I am just trying out an architecture which has daily solr servers at the lowest level followed by aggregators at each level. Hence, an aggregator node does not have data of its own. The problem is while the L1aggregator seem to work perfectly fine, the L2Aggregator seem to throw internal server error (stack trace posted below). I specify the address of shards in the aggregator's config file. Also, all the documents have a unique ids. While I was trying to debug the error, I noticed that a L2 aggregator solr makes two calls to L1 aggregator: one to get the docs' ids and the other to get data for thos doc ids. While the first call works, the second call seems to throw the error pasted below. Any help would be highly appreciated. Thanks HTTP Status 500 - Internal Server Error Internal Server Error request: http://l1aggregator:8080/solr/core-00/selectorg.apache.solr.common.SolrException: Internal Server Error Internal Server Error request: http://l1aggregator:8080/solr/core-00/select at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:427) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:249) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:129) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:103) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) -- *type* Status report *message* *Internal Server Error Internal Server Error request: http://l1aggregator:8080/solr/core-00/selectorg.apache.solr.common.SolrException: Internal Server Error Internal Server Error request: http://**l1aggregator**:8080/solr/core-00/select at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:427) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:249) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:129) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:103) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)* *description* *The server encountered an internal error (Internal Server Error Internal Server Error request: http://**l1aggregator**:8080/solr/core-00/select org.apache.solr.common.SolrException: Internal Server Error Internal Server Error request: http://**l1aggregator**:8080/solr/core-00/select at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:427) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:249) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:129) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:103) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at
Re: Solr ids query parameter
well it's an interesting solution but my use case is such that any of the daily servers could hold the data for a query and hence i need an aggregator to sum up the results. The only issue is, an aggregator could sum up only certain number (I heard around 200) of cores. Hence I would need to manually sum results from different aggregators if for some reason my query spans more than 200 cores. Digging more into the issue I found that solr does distributed searching in two steps. First it queries for doc ids and then gets the data corresponding to those doc ids. I am seeing the error when it is trying to compare the score of two documents. Below is a better stack trace and its similar to https://issues.apache.org/jira/browse/SOLR-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel . SEVERE: java.lang.NullPointerException at org.apache.solr.handler.component.ShardFieldSortedHitQueue$1.compare(ShardDoc.java:236) at org.apache.solr.handler.component.ShardFieldSortedHitQueue.lessThan(ShardDoc.java:160) at org.apache.solr.handler.component.ShardFieldSortedHitQueue.lessThan(ShardDoc.java:101) at org.apache.lucene.util.PriorityQueue.upHeap(PriorityQueue.java:223) at org.apache.lucene.util.PriorityQueue.add(PriorityQueue.java:132) at org.apache.lucene.util.PriorityQueue.insertWithOverflow(PriorityQueue.java:148) at org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:786) at org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:587) at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:566) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:283) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:309) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) On Thu, Jul 12, 2012 at 2:46 PM, Michael Della Bitta michael.della.bi...@appinions.com wrote: I'm pretty sure a sharded query/response gets tagged in a special way that might be incompatible with multiple tiers like this. Had you considered a single tier that consisted of a load-balanced pool of 'aggregator' servers? Michael Della Bitta Appinions, Inc. -- Where Influence Isn’t a Game. http://www.appinions.com On Thu, Jul 12, 2012 at 5:36 PM, Peter Markey sudoma...@gmail.com wrote: Hello, I am trying to do a distributed search with solr and for some reason I get an internal server error. The set up is like this: I have 4 solr servers that index data (say daily each with 10 cores) and I use another bunch of solr instances (lets call one of them as L1aggregator) that does a distributed request to all the 40 cores of 4 solr servers. I also have another solr instance (lets call it L2aggregator) that would be collecting data from multiple L1aggregator solr instances. Its like a multi-level structure. I am just trying out an architecture which has daily solr servers at the lowest level followed by aggregators at each level. Hence, an aggregator node does not have data of its own. The problem is while the L1aggregator seem to work perfectly fine, the L2Aggregator seem to throw internal server error (stack trace posted below). I specify the address of shards in the aggregator's config file. Also, all the documents have a unique ids. While I was trying to debug the error, I noticed
Re: query syntax to find ??? chars
When you put a backslash in a Java string you need to escape it with another backslash: content:\\?\\?\\?\\?\\?\\? -- Jack Krupansky -Original Message- From: Alexander Aristov Sent: Thursday, July 12, 2012 8:30 AM To: solr-user@lucene.apache.org Subject: Re: query syntax to find ??? chars don't know why but doesn't work. :( Best Regards Alexander Aristov On 11 July 2012 23:54, Yury Kats yuryk...@yahoo.com wrote: On 7/11/2012 2:55 PM, Alexander Aristov wrote: content:?? doesn't work :) I would try escaping them: content:\?\?\?\?\?\?
question on joins related to noSQL
I currently have this example test case working against lucene RAMDirectory. very simple test case... https://github.com/deanhiller/nosqlORM/blob/memoryindexer/input/javasrc/com/alvazan/test/TestIndexAndOrParens.java with this Entity and his namedQueries https://github.com/deanhiller/nosqlORM/blob/memoryindexer/input/javasrc/com/alvazan/test/db/Activity.java We allow applications to split their 1 billion/1 trillion, whatever into 1 million or 1 billion or whatever number of indexes and dba's can query their partitions by grabbing the index and running standard SQL. Currently, yes, we translate from SQL to lucene query language. This has really just been a playground and now I would like to start doing joins. 1. I was wondering though, does SOLR only support a join of two indexes or can you join more than two indexes? 2. Also, anyone know if solandra has support for this? Currently, we actually cache all writes and only send them in the flush so if any additions to the index fail earlier nothing was written so there is a better chance of data being consistent. thanks, Dean
Index version on slave incrementing to higher than master
Hi,
I am running solr 1.4.0+ds1-1ubuntu1. I have a master server that has a
number of solr instances running on it (150 or so), and nightly most of
them have documents written to them. The script that does these writes
(adds) does a commit and an optimize on the indexes when it's entirely
finished updating them, then initiates replication on the slave per
instance. In this configuration, the index versions between master and
slave remain in synch.
The optimize portion, which, again, happens nightly, is taking a lot of
time and I think it's unnecessary. I was hoping to stop doing this explicit
optimize, and to let my merge policy handle that. However, if I don't do an
optimize, and only do a commit before initiating slave replication, some
hours later the slave is, for reasons that are unclear to me, incrementing
its index version to 1 higher than the master.
I am not really sure I understand the logs, but it looks like the
incremented index version is the result of an optimize on the slave, but I
am never issuing any commands against the slave aside from initiating
replication, and I don't think there's anything in my solr configuration
that would be initiating this. I do have autoCommit on with maxDocs of
1000, but since I am initiating slave replication after doing a commit on
the master, I don't think there would ever be any uncommitted documents on
the slave. I do have a merge policy configured, but it's not clear to me
that it has anything to do with this. And if it did, I'd expect to see
similar behavior on the master (right?).
I have included a snipped from my slave logs that shows this issue. In this
snipped index version 1286065171264 is what the master has,
and 1286065171265 is what the slave increments itself to, which is then out
of synch with the master in terms of version numbers. Nothing that I know
of is issuing any commands to the slave at this time. If I understand these
logs (I might not), it looks like something issued an optimize that took
1023720ms? Any ideas?
Thanks in advance.
Andy
Jul 12, 2012 12:21:14 PM org.apache.solr.update.SolrIndexWriter close
FINE: Closing Writer DirectUpdateHandler2
Jul 12, 2012 12:21:14 PM org.apache.solr.core.SolrDeletionPolicy onCommit
INFO: SolrDeletionPolicy.onCommit: commits:num=2
commit{dir=/var/lib/ontolo/solr/o_3952/index,segFN=segments_h8,version=1286065171264,generation=620,filenames=[_h6.fnm,
_h5.nrm, segments_h8, _h4.nrm, _h5.tii, _h4
.tii, _h5.tis, _h4.tis, _h4.fdx, _h5.fnm, _h6.tii, _h4.fdt, _h5.fdt,
_h5.fdx, _h5.frq, _h4.fnm, _h6.frq, _h6.tis, _h4.prx, _h4.frq, _h6.nrm,
_h5.prx, _h6.prx, _h6.fdt, _h6
.fdx]
commit{dir=/var/lib/ontolo/solr/o_3952/index,segFN=segments_h9,version=1286065171265,generation=621,filenames=[_h7.tis,
_h7.fdx, _h7.fnm, _h7.fdt, _h7.prx, segment
s_h9, _h7.nrm, _h7.tii, _h7.frq]
Jul 12, 2012 12:21:14 PM org.apache.solr.core.SolrDeletionPolicy
updateCommits
INFO: newest commit = 1286065171265
Jul 12, 2012 12:21:14 PM org.apache.solr.search.SolrIndexSearcher init
INFO: Opening Searcher@4ac62082 main
Jul 12, 2012 12:21:14 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: end_commit_flush
Jul 12, 2012 12:21:14 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@4ac62082 main from Searcher@48d901f7 main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative
_inserts=0,cumulative_evictions=0}
Jul 12, 2012 12:21:14 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@4ac62082 main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jul 12, 2012 12:21:14 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@4ac62082 main from Searcher@48d901f7 main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=18,cumulative_hits=14,cumulative_hitratio=0.77,cumulative_inserts=4,cumulative_evictions=0}
Jul 12, 2012 12:21:14 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@4ac62082 main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=18,cumulative_hits=14,cumulative_hitratio=0.77,cumulative_inserts=4,cumulative_evictions=0}
Jul 12, 2012 12:21:14 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@4ac62082 main from Searcher@48d901f7 main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=150,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=69,cumulative_evictions=0}
Jul 12, 2012 12:21:14 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@4ac62082 main
Re: Updating documents
On Thu, Jul 12, 2012 at 3:20 PM, Jonatan Fournier jonatan.fourn...@gmail.com wrote: Is there a flag for: if document does not exist, create it for me? Not currently, but it certainly makes sense. The implementation should be easy. The most difficult part is figuring out the best syntax to specify this. Another idea: we could possibly switch to create-if-not-exist by default, and use the existing optimistic concurrency mechanism to specify that the document should exist. So specify _version_=1 if the document should exist and _version_=0 (the default) if you don't care. -Yonik http://lucidimagination.com
Re: Solr facet multiple constraint
Ok well i know about the complexity that i can put into fq with AND and OR conditions but at the moment when i put fq=user:10,facet.field=user, the query returns me all the facets not taking into account the fq=user:10 that's the problem. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-facet-multiple-constraint-tp3992974p3994783.html Sent from the Solr - User mailing list archive at Nabble.com.
Email keeps bouncing
Whenever I reply to an email to this list I got a failure notice (please see below) What can I do? Sorry, we were unable to deliver your message to the following address. solr-user@lucene.apache.org: Remote host said: 552 spam score (6.0) exceeded threshold (FREEMAIL_FORGED_REPLYTO,FSL_FREEMAIL_1,FSL_FREEMAIL_2,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,URI_HEX ) [BODY]
