Re: Cannot get uf (user fields) to work
Hi Emir, Like I replied to Jack, I don't get it. I need to see some concrete examples. Thanks Steve On Thu, May 14, 2015 at 8:30 AM, Emir Arnautovic emir.arnauto...@sematext.com wrote: Hi Steve, No - you should use qf to limit search to fields you want. If you do not plan to allow users to explicitly state fields they want to search, you don't have to use uf parameter. So, drop uf, and set qf to F3. Thanks, Emir On 14.05.2015 14:09, Steven White wrote: Hi Emir, Are you saying I should remove qf if I want to use uf? I just tried that and got exception: java.lang.RuntimeException at org.apache.solr.search.ExtendedDismaxQParser$ExtendedDismaxConfiguration.init(ExtendedDismaxQParser.java:1497) at org.apache.solr.search.ExtendedDismaxQParser.createConfiguration(ExtendedDismaxQParser.java:270) at org.apache.solr.search.ExtendedDismaxQParser.init(ExtendedDismaxQParser.java:132) at org.apache.solr.search.ExtendedDismaxQParserPlugin.createParser(ExtendedDismaxQParserPlugin.java:37) at org.apache.solr.search.QParser.getParser(QParser.java:315) at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:156) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:196) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984) at Thanks Steve On Thu, May 14, 2015 at 8:00 AM, Emir Arnautovic emir.arnauto...@sematext.com wrote: Hi Steve, You use qf to limit search on some fields. Thanks, Emir On 14.05.2015 13:49, Steven White wrote: Hi Everyone, I'm trying to utilize uf but it doesn't work. My reading of it per: https://cwiki.apache.org/confluence/display/solr/The+Extended+DisMax+Query+Parser is I can use it to limit into which fields a search is performed. E.g.: If my index has F1, F2, F3, F4, and I set uf to F3, than I should get hits on F3 only. Here is my request handler: requestHandler name=/select_group_a class=solr.SearchHandler lst name=defaults str name=echoParamsexplicit/str int name=rows20/int str name=defTypeedismax/str str name=qfF1 F2 F3 F4/str str name=ufF3/str str name=fltype,id,score/str str name=wtxml/str str name=indenttrue/str /lst /requestHandler I also tried passing uf on the URL and it too didn't work. I'm getting hits on terms that are in other fields. Is my usage of uf wrong? Thanks Steve -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/
Re: Cannot get uf (user fields) to work
Steve, Your config should be like: requestHandler name=/select_group_a class=solr.SearchHandler lst name=defaults str name=echoParamsexplicit/str int name=rows20/int str name=defTypeedismax/str str name=qfF3/str str name=fltype,id,score/str str name=wtxml/str str name=indenttrue/str /lst /requestHandler This means when you query with something it will be equivalent to F3:something if you use standard parser. If you specify uf, than it will allow query to contain that field i.e. even you said in qf=F1 F2 F3, if you have in uf=F1, you will be able to define q=F1:something and it will search only F1. Otherwise it will not - did not use it so not sure if it will throw parsing error or will threat F1:something as query and search it in F1, F2 and F3. Thanks, Emir On 14.05.2015 14:37, Steven White wrote: Hi Emir, Like I replied to Jack, I don't get it. I need to see some concrete examples. Thanks Steve On Thu, May 14, 2015 at 8:30 AM, Emir Arnautovic emir.arnauto...@sematext.com wrote: Hi Steve, No - you should use qf to limit search to fields you want. If you do not plan to allow users to explicitly state fields they want to search, you don't have to use uf parameter. So, drop uf, and set qf to F3. Thanks, Emir On 14.05.2015 14:09, Steven White wrote: Hi Emir, Are you saying I should remove qf if I want to use uf? I just tried that and got exception: java.lang.RuntimeException at org.apache.solr.search.ExtendedDismaxQParser$ExtendedDismaxConfiguration.init(ExtendedDismaxQParser.java:1497) at org.apache.solr.search.ExtendedDismaxQParser.createConfiguration(ExtendedDismaxQParser.java:270) at org.apache.solr.search.ExtendedDismaxQParser.init(ExtendedDismaxQParser.java:132) at org.apache.solr.search.ExtendedDismaxQParserPlugin.createParser(ExtendedDismaxQParserPlugin.java:37) at org.apache.solr.search.QParser.getParser(QParser.java:315) at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:156) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:196) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984) at Thanks Steve On Thu, May 14, 2015 at 8:00 AM, Emir Arnautovic emir.arnauto...@sematext.com wrote: Hi Steve, You use qf to limit search on some fields. Thanks, Emir On 14.05.2015 13:49, Steven White wrote: Hi Everyone, I'm trying to utilize uf but it doesn't work. My reading of it per: https://cwiki.apache.org/confluence/display/solr/The+Extended+DisMax+Query+Parser is I can use it to limit into which fields a search is performed. E.g.: If my index has F1, F2, F3, F4, and I set uf to F3, than I should get hits on F3 only. Here is my request handler: requestHandler name=/select_group_a class=solr.SearchHandler lst name=defaults str name=echoParamsexplicit/str int name=rows20/int str name=defTypeedismax/str str name=qfF1 F2 F3 F4/str str name=ufF3/str str name=fltype,id,score/str str name=wtxml/str str name=indenttrue/str /lst /requestHandler I also tried passing uf on the URL and it too didn't work. I'm getting hits on terms that are in other fields. Is my usage of uf wrong? Thanks Steve -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/
Re: Transactional Behavior
On 5/12/2015 10:58 AM, Amr Ali wrote: I have a business case in which I need to be able for the rollback. When I tried add/commit I was not able to prevent other threads that write to a given Solr core from committing everything. I also tried indexwriter but Solr did not get changes until we restart it. This is how Lucene is designed. The bulk of Solr's functionality is provided directly by Lucene, including the way that commits work. Adding transaction support with individual rollback capability would be a VERY fundamental and low-level change to Lucene. I bet you would find that major pieces of Lucene functionality are dependent on the current non-transactional design for commits, which means that changing it would probably require changes in every major subsystem found in the code, and that general performance might be very adversely affected. Verifying correctness and fixing problems could take weeks or months. I don't think the benefit is worth the pain, especially when there's a simple workaround -- don't overlap changes that need to be treated as different transactions -- do them at different times. Thanks, Shawn
Re: Real-Time get and Dynamic Fields: possible bug.
Ehem, *_target --- *_facet.
2015-05-14 16:47 GMT+02:00 Luis Cappa Banda luisca...@gmail.com:
Hi Yonik,
Yes, they are the target from copyFields in the schema.xml. This *_target
fields are suposed to be used in some specific searchable (thus, tokenized)
fields that in the future are candidates to be faceted to return some
stats. For example, imagine that you have a field storing a directory path
and you want to search by. Also, you may want to facet by the whole
directory path value (not just their terms). Thats why I' m storing both
field values: searchable and tokenized one, string and 'facet candidate'
one.
What I do not understand is that both i18n* and *_target are dynamic,
indexed and stored values. The only difference is that *_target one is
multivalued. Does it have some sense?
Regards
- Luis Cappa
2015-05-14 16:42 GMT+02:00 Yonik Seeley ysee...@gmail.com:
Are the _facet fields the target of a copyField in the schema?
Realtime get either gets the values from the transaction log (and if
you didn't send it the values, they won't be there) or gets them from
the index to try and reconstruct what was sent in.
It's generally not recommended to have copyField targets stored, or
have a mix of explicitly set values and copyField values in the same
field.
-Yonik
On Thu, May 14, 2015 at 7:17 AM, Luis Cappa Banda luisca...@gmail.com
wrote:
Hi there,
I have the following dynamicFields definition in my schema.xml:
!-- I18n DynamicFields --
dynamicField name=i18n* type=string indexed=true stored=true
/ !--
DynamicFields used typically for faceting issues by copying values from
other existing fields-- dynamicField name=*_facet type=string
indexed=
true stored=true multiValued=true /
I' ve seen that when fetching documents with /select?q=id:whateverId,
the
results returned include both i18n* and *_facet fields filled. However,
when using real-time request handler (/get?ids:whateverIds) the result
fetched include only i18n* dynamic fields, but *_facet ones are not
included.
I have the impression during /get RequestHandler the server-side regular
expression used when parsing fields and fields values to return
documents
with existing dynamic fields seems to be wrong. From the client side,
I' ve
checked that the class DocField.java that parses SolrDocument to Bean
ones
uses the following matcher:
} else if (annotation.value().indexOf('*') = 0) { // dynamic fields
are
annotated as @Field(categories_*)
// if the field was annotated as a dynamic field, convert the name into
a
pattern
// the wildcard (*) is supposed to be either a prefix or a suffix, hence
the use of replaceFirst
name = annotation.value().replaceFirst(\\*, \\.*);
dynamicFieldNamePatternMatcher = Pattern.compile(^ + name + $);
} else {
name = annotation.value();
}
So maybe a similar behavior from the server-side is wrong. That' s the
only
reason I find to understand why when using /select all fields are
returned
but when using /get those that matches *_facet regexp are not.
If you can confirm that this is a bug (because maybe is the expected
behavior, but after some years using Solr I think it is not) I can
create
the JIRA issue and debug it more deeply to apply a patch with the aim to
help.
Regards,
--
- Luis Cappa
--
- Luis Cappa
--
- Luis Cappa
Re: Real-Time get and Dynamic Fields: possible bug.
Hi Yonik,
Yes, they are the target from copyFields in the schema.xml. This *_target
fields are suposed to be used in some specific searchable (thus, tokenized)
fields that in the future are candidates to be faceted to return some
stats. For example, imagine that you have a field storing a directory path
and you want to search by. Also, you may want to facet by the whole
directory path value (not just their terms). Thats why I' m storing both
field values: searchable and tokenized one, string and 'facet candidate'
one.
What I do not understand is that both i18n* and *_target are dynamic,
indexed and stored values. The only difference is that *_target one is
multivalued. Does it have some sense?
Regards
- Luis Cappa
2015-05-14 16:42 GMT+02:00 Yonik Seeley ysee...@gmail.com:
Are the _facet fields the target of a copyField in the schema?
Realtime get either gets the values from the transaction log (and if
you didn't send it the values, they won't be there) or gets them from
the index to try and reconstruct what was sent in.
It's generally not recommended to have copyField targets stored, or
have a mix of explicitly set values and copyField values in the same
field.
-Yonik
On Thu, May 14, 2015 at 7:17 AM, Luis Cappa Banda luisca...@gmail.com
wrote:
Hi there,
I have the following dynamicFields definition in my schema.xml:
!-- I18n DynamicFields --
dynamicField name=i18n* type=string indexed=true stored=true /
!--
DynamicFields used typically for faceting issues by copying values from
other existing fields-- dynamicField name=*_facet type=string
indexed=
true stored=true multiValued=true /
I' ve seen that when fetching documents with /select?q=id:whateverId, the
results returned include both i18n* and *_facet fields filled. However,
when using real-time request handler (/get?ids:whateverIds) the result
fetched include only i18n* dynamic fields, but *_facet ones are not
included.
I have the impression during /get RequestHandler the server-side regular
expression used when parsing fields and fields values to return documents
with existing dynamic fields seems to be wrong. From the client side, I'
ve
checked that the class DocField.java that parses SolrDocument to Bean
ones
uses the following matcher:
} else if (annotation.value().indexOf('*') = 0) { // dynamic fields are
annotated as @Field(categories_*)
// if the field was annotated as a dynamic field, convert the name into a
pattern
// the wildcard (*) is supposed to be either a prefix or a suffix, hence
the use of replaceFirst
name = annotation.value().replaceFirst(\\*, \\.*);
dynamicFieldNamePatternMatcher = Pattern.compile(^ + name + $);
} else {
name = annotation.value();
}
So maybe a similar behavior from the server-side is wrong. That' s the
only
reason I find to understand why when using /select all fields are
returned
but when using /get those that matches *_facet regexp are not.
If you can confirm that this is a bug (because maybe is the expected
behavior, but after some years using Solr I think it is not) I can create
the JIRA issue and debug it more deeply to apply a patch with the aim to
help.
Regards,
--
- Luis Cappa
--
- Luis Cappa
RE: Transactional Behavior
Hello guys, Any expert advice this issue? -- Regards, Amr Ali City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt Ext: 278 -Original Message- From: Amr Ali Sent: Wednesday, May 13, 2015 10:58 AM To: solr-user@lucene.apache.org Subject: RE: Transactional Behavior Hello Emir, But this is not a transaction because if some of the bulk I need to add is committed; they will be searchable. In a transaction I need to insert a bulk of data (all bulk data will be searchable once) or roll it back according to some business scenarios. -- Regards, Amr Ali City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt Ext: 278 -Original Message- From: Emir Arnautovic [mailto:emir.arnauto...@sematext.com] Sent: Tuesday, May 12, 2015 10:46 PM To: solr-user@lucene.apache.org Subject: Re: Transactional Behavior Hi Amr, One option is to include transaction id in your documents and do delete in case of failed transaction. It is not cheap option - additional field if you don't have something to use to identify transaction. Assuming rollback will not happen to often deleting is not that big issue. Thanks, Emir -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On 12.05.2015 22:37, Amr Ali wrote: Please check this https://lucene.apache.org/solr/4_1_0/solr-solrj/org/apache/solr/client /solrj/SolrServer.html#rollback() Note that this is not a true rollback as in databases. Content you have previously added may have been committed due to autoCommit, buffer full, other client performing a commit etc. It is not a real rollback if you have two threads T1 and T2 that are adding. If T1 is adding 500 and T2 is adding 3 then T2 will commit its 3 document PLUS the documents added by T1 (because T2 will finish add/commit before T2 due to the documents number). Solr transactions are server side only. -- Regards, Amr Ali City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt Ext: 278 -Original Message- From: Jack Krupansky [mailto:jack.krupan...@gmail.com] Sent: Tuesday, May 12, 2015 10:24 PM To: solr-user@lucene.apache.org Subject: Re: Transactional Behavior Solr does have a rollback/ command, but it is an expert feature and not so clear how it works in SolrCloud. See: https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+I ndex+Handlers and https://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22 -- Jack Krupansky On Tue, May 12, 2015 at 12:58 PM, Amr Ali amr_...@siliconexpert.com wrote: Hello, I have a business case in which I need to be able for the rollback. When I tried add/commit I was not able to prevent other threads that write to a given Solr core from committing everything. I also tried indexwriter but Solr did not get changes until we restart it. -- Regards, Amr Ali City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt Ext: 278 -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/
Re: Cannot get uf (user fields) to work
Hi Steve, No - you should use qf to limit search to fields you want. If you do not plan to allow users to explicitly state fields they want to search, you don't have to use uf parameter. So, drop uf, and set qf to F3. Thanks, Emir On 14.05.2015 14:09, Steven White wrote: Hi Emir, Are you saying I should remove qf if I want to use uf? I just tried that and got exception: java.lang.RuntimeException at org.apache.solr.search.ExtendedDismaxQParser$ExtendedDismaxConfiguration.init(ExtendedDismaxQParser.java:1497) at org.apache.solr.search.ExtendedDismaxQParser.createConfiguration(ExtendedDismaxQParser.java:270) at org.apache.solr.search.ExtendedDismaxQParser.init(ExtendedDismaxQParser.java:132) at org.apache.solr.search.ExtendedDismaxQParserPlugin.createParser(ExtendedDismaxQParserPlugin.java:37) at org.apache.solr.search.QParser.getParser(QParser.java:315) at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:156) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:196) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984) at Thanks Steve On Thu, May 14, 2015 at 8:00 AM, Emir Arnautovic emir.arnauto...@sematext.com wrote: Hi Steve, You use qf to limit search on some fields. Thanks, Emir On 14.05.2015 13:49, Steven White wrote: Hi Everyone, I'm trying to utilize uf but it doesn't work. My reading of it per: https://cwiki.apache.org/confluence/display/solr/The+Extended+DisMax+Query+Parser is I can use it to limit into which fields a search is performed. E.g.: If my index has F1, F2, F3, F4, and I set uf to F3, than I should get hits on F3 only. Here is my request handler: requestHandler name=/select_group_a class=solr.SearchHandler lst name=defaults str name=echoParamsexplicit/str int name=rows20/int str name=defTypeedismax/str str name=qfF1 F2 F3 F4/str str name=ufF3/str str name=fltype,id,score/str str name=wtxml/str str name=indenttrue/str /lst /requestHandler I also tried passing uf on the URL and it too didn't work. I'm getting hits on terms that are in other fields. Is my usage of uf wrong? Thanks Steve -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/
Re: Cannot get uf (user fields) to work
One example: https://github.com/arafalov/solr-indexing-book/blob/master/published/languages/conf/solrconfig.xml#L36 This is from my book: https://www.packtpub.com/big-data-and-business-intelligence/instant-apache-solr-indexing-data-how-instant But I am not sure if it affects the specific concern you have, hopefully other emails will. Regards, Alex. Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter: http://www.solr-start.com/ On 14 May 2015 at 22:37, Steven White swhite4...@gmail.com wrote: Hi Emir, Like I replied to Jack, I don't get it. I need to see some concrete examples. Thanks Steve
Re: Cannot get uf (user fields) to work
The uf parameter applies to the user query string, the q parameter - the q parameter may only references fields from uf in the q string. The qf parameter specifies fields to be queried behind the scenes and is not seen by the user. The term user refers to the person who is specifying the text that goes into the q parameter, not the client app that is forming the overall Solr query request. -- Jack Krupansky On Thu, May 14, 2015 at 7:49 AM, Steven White swhite4...@gmail.com wrote: Hi Everyone, I'm trying to utilize uf but it doesn't work. My reading of it per: https://cwiki.apache.org/confluence/display/solr/The+Extended+DisMax+Query+Parser is I can use it to limit into which fields a search is performed. E.g.: If my index has F1, F2, F3, F4, and I set uf to F3, than I should get hits on F3 only. Here is my request handler: requestHandler name=/select_group_a class=solr.SearchHandler lst name=defaults str name=echoParamsexplicit/str int name=rows20/int str name=defTypeedismax/str str name=qfF1 F2 F3 F4/str str name=ufF3/str str name=fltype,id,score/str str name=wtxml/str str name=indenttrue/str /lst /requestHandler I also tried passing uf on the URL and it too didn't work. I'm getting hits on terms that are in other fields. Is my usage of uf wrong? Thanks Steve
Re: Cannot get uf (user fields) to work
On 5/14/2015 6:33 AM, Steven White wrote: I don't follow. Can you give me an example on how and when to use uf? The uf parameter controls which fields are allowed in the query string. If you have uf=f1 f2 then q=f3:foo won't work the way you would expect. If f3 is in the qf parameter, then it will still be used by the parser. I tried the equivalent of q=f3:foo on my own index with uf set to fields that did not include that field. When a field is disallowed, but used in the query, there is no error message. Instead, the data is interpreted as raw input, so that example would work like q=f3\:foo instead. The colon becomes part of the actual query text. Testing shows that uf does *NOT* affect filter queries, only the main query. I think this is because filter queries default to the lucene parser, not the parser specified in defType. Thanks, Shawn
Re: Real-Time get and Dynamic Fields: possible bug.
Are the _facet fields the target of a copyField in the schema?
Realtime get either gets the values from the transaction log (and if
you didn't send it the values, they won't be there) or gets them from
the index to try and reconstruct what was sent in.
It's generally not recommended to have copyField targets stored, or
have a mix of explicitly set values and copyField values in the same
field.
-Yonik
On Thu, May 14, 2015 at 7:17 AM, Luis Cappa Banda luisca...@gmail.com wrote:
Hi there,
I have the following dynamicFields definition in my schema.xml:
!-- I18n DynamicFields --
dynamicField name=i18n* type=string indexed=true stored=true / !--
DynamicFields used typically for faceting issues by copying values from
other existing fields-- dynamicField name=*_facet type=string indexed=
true stored=true multiValued=true /
I' ve seen that when fetching documents with /select?q=id:whateverId, the
results returned include both i18n* and *_facet fields filled. However,
when using real-time request handler (/get?ids:whateverIds) the result
fetched include only i18n* dynamic fields, but *_facet ones are not
included.
I have the impression during /get RequestHandler the server-side regular
expression used when parsing fields and fields values to return documents
with existing dynamic fields seems to be wrong. From the client side, I' ve
checked that the class DocField.java that parses SolrDocument to Bean ones
uses the following matcher:
} else if (annotation.value().indexOf('*') = 0) { // dynamic fields are
annotated as @Field(categories_*)
// if the field was annotated as a dynamic field, convert the name into a
pattern
// the wildcard (*) is supposed to be either a prefix or a suffix, hence
the use of replaceFirst
name = annotation.value().replaceFirst(\\*, \\.*);
dynamicFieldNamePatternMatcher = Pattern.compile(^ + name + $);
} else {
name = annotation.value();
}
So maybe a similar behavior from the server-side is wrong. That' s the only
reason I find to understand why when using /select all fields are returned
but when using /get those that matches *_facet regexp are not.
If you can confirm that this is a bug (because maybe is the expected
behavior, but after some years using Solr I think it is not) I can create
the JIRA issue and debug it more deeply to apply a patch with the aim to
help.
Regards,
--
- Luis Cappa
Re: Cannot get uf (user fields) to work
Hi Emir, Are you saying I should remove qf if I want to use uf? I just tried that and got exception: java.lang.RuntimeException at org.apache.solr.search.ExtendedDismaxQParser$ExtendedDismaxConfiguration.init(ExtendedDismaxQParser.java:1497) at org.apache.solr.search.ExtendedDismaxQParser.createConfiguration(ExtendedDismaxQParser.java:270) at org.apache.solr.search.ExtendedDismaxQParser.init(ExtendedDismaxQParser.java:132) at org.apache.solr.search.ExtendedDismaxQParserPlugin.createParser(ExtendedDismaxQParserPlugin.java:37) at org.apache.solr.search.QParser.getParser(QParser.java:315) at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:156) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:196) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984) at Thanks Steve On Thu, May 14, 2015 at 8:00 AM, Emir Arnautovic emir.arnauto...@sematext.com wrote: Hi Steve, You use qf to limit search on some fields. Thanks, Emir On 14.05.2015 13:49, Steven White wrote: Hi Everyone, I'm trying to utilize uf but it doesn't work. My reading of it per: https://cwiki.apache.org/confluence/display/solr/The+Extended+DisMax+Query+Parser is I can use it to limit into which fields a search is performed. E.g.: If my index has F1, F2, F3, F4, and I set uf to F3, than I should get hits on F3 only. Here is my request handler: requestHandler name=/select_group_a class=solr.SearchHandler lst name=defaults str name=echoParamsexplicit/str int name=rows20/int str name=defTypeedismax/str str name=qfF1 F2 F3 F4/str str name=ufF3/str str name=fltype,id,score/str str name=wtxml/str str name=indenttrue/str /lst /requestHandler I also tried passing uf on the URL and it too didn't work. I'm getting hits on terms that are in other fields. Is my usage of uf wrong? Thanks Steve -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/
Re: Cannot get uf (user fields) to work
Hi Jack, I don't follow. Can you give me an example on how and when to use uf? Thanks, Steve On Thu, May 14, 2015 at 8:15 AM, Jack Krupansky jack.krupan...@gmail.com wrote: The uf parameter applies to the user query string, the q parameter - the q parameter may only references fields from uf in the q string. The qf parameter specifies fields to be queried behind the scenes and is not seen by the user. The term user refers to the person who is specifying the text that goes into the q parameter, not the client app that is forming the overall Solr query request. -- Jack Krupansky On Thu, May 14, 2015 at 7:49 AM, Steven White swhite4...@gmail.com wrote: Hi Everyone, I'm trying to utilize uf but it doesn't work. My reading of it per: https://cwiki.apache.org/confluence/display/solr/The+Extended+DisMax+Query+Parser is I can use it to limit into which fields a search is performed. E.g.: If my index has F1, F2, F3, F4, and I set uf to F3, than I should get hits on F3 only. Here is my request handler: requestHandler name=/select_group_a class=solr.SearchHandler lst name=defaults str name=echoParamsexplicit/str int name=rows20/int str name=defTypeedismax/str str name=qfF1 F2 F3 F4/str str name=ufF3/str str name=fltype,id,score/str str name=wtxml/str str name=indenttrue/str /lst /requestHandler I also tried passing uf on the URL and it too didn't work. I'm getting hits on terms that are in other fields. Is my usage of uf wrong? Thanks Steve
Re: Transactional Behavior
Hi Amr, As far as I am aware, SOLR does not support transaction isolation. If you really need it, you will have to isolate it yourself. Easiest way is to sacrifice concurrent updates. Other way of achieving it would be to to index in separate indices (maybe in memory) concurrently and than merge it with your main index in isolation. There are probably better solutions but this is from the top of my head. Thanks, Emir On 14.05.2015 14:21, Amr Ali wrote: Hello guys, Any expert advice this issue? -- Regards, Amr Ali City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt Ext: 278 -Original Message- From: Amr Ali Sent: Wednesday, May 13, 2015 10:58 AM To: solr-user@lucene.apache.org Subject: RE: Transactional Behavior Hello Emir, But this is not a transaction because if some of the bulk I need to add is committed; they will be searchable. In a transaction I need to insert a bulk of data (all bulk data will be searchable once) or roll it back according to some business scenarios. -- Regards, Amr Ali City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt Ext: 278 -Original Message- From: Emir Arnautovic [mailto:emir.arnauto...@sematext.com] Sent: Tuesday, May 12, 2015 10:46 PM To: solr-user@lucene.apache.org Subject: Re: Transactional Behavior Hi Amr, One option is to include transaction id in your documents and do delete in case of failed transaction. It is not cheap option - additional field if you don't have something to use to identify transaction. Assuming rollback will not happen to often deleting is not that big issue. Thanks, Emir -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On 12.05.2015 22:37, Amr Ali wrote: Please check this https://lucene.apache.org/solr/4_1_0/solr-solrj/org/apache/solr/client /solrj/SolrServer.html#rollback() Note that this is not a true rollback as in databases. Content you have previously added may have been committed due to autoCommit, buffer full, other client performing a commit etc. It is not a real rollback if you have two threads T1 and T2 that are adding. If T1 is adding 500 and T2 is adding 3 then T2 will commit its 3 document PLUS the documents added by T1 (because T2 will finish add/commit before T2 due to the documents number). Solr transactions are server side only. -- Regards, Amr Ali City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt Ext: 278 -Original Message- From: Jack Krupansky [mailto:jack.krupan...@gmail.com] Sent: Tuesday, May 12, 2015 10:24 PM To: solr-user@lucene.apache.org Subject: Re: Transactional Behavior Solr does have a rollback/ command, but it is an expert feature and not so clear how it works in SolrCloud. See: https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+I ndex+Handlers and https://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22 -- Jack Krupansky On Tue, May 12, 2015 at 12:58 PM, Amr Ali amr_...@siliconexpert.com wrote: Hello, I have a business case in which I need to be able for the rollback. When I tried add/commit I was not able to prevent other threads that write to a given Solr core from committing everything. I also tried indexwriter but Solr did not get changes until we restart it. -- Regards, Amr Ali City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt Ext: 278 -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/
Re: Transactional Behavior
Solr does not do transactions at all. The “commit” is to signal the end of a batch load and start the processing. You can hack some things, but there is no way to roll back a delete or a update. If you need this, you need to use a database, not a search engine. I recommend MarkLogic. It is a full-featured database that includes search features. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) On May 14, 2015, at 6:12 AM, Emir Arnautovic emir.arnauto...@sematext.com wrote: Hi Amr, As far as I am aware, SOLR does not support transaction isolation. If you really need it, you will have to isolate it yourself. Easiest way is to sacrifice concurrent updates. Other way of achieving it would be to to index in separate indices (maybe in memory) concurrently and than merge it with your main index in isolation. There are probably better solutions but this is from the top of my head. Thanks, Emir On 14.05.2015 14:21, Amr Ali wrote: Hello guys, Any expert advice this issue? -- Regards, Amr Ali City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt Ext: 278 -Original Message- From: Amr Ali Sent: Wednesday, May 13, 2015 10:58 AM To: solr-user@lucene.apache.org Subject: RE: Transactional Behavior Hello Emir, But this is not a transaction because if some of the bulk I need to add is committed; they will be searchable. In a transaction I need to insert a bulk of data (all bulk data will be searchable once) or roll it back according to some business scenarios. -- Regards, Amr Ali City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt Ext: 278 -Original Message- From: Emir Arnautovic [mailto:emir.arnauto...@sematext.com] Sent: Tuesday, May 12, 2015 10:46 PM To: solr-user@lucene.apache.org Subject: Re: Transactional Behavior Hi Amr, One option is to include transaction id in your documents and do delete in case of failed transaction. It is not cheap option - additional field if you don't have something to use to identify transaction. Assuming rollback will not happen to often deleting is not that big issue. Thanks, Emir -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On 12.05.2015 22:37, Amr Ali wrote: Please check this https://lucene.apache.org/solr/4_1_0/solr-solrj/org/apache/solr/client /solrj/SolrServer.html#rollback() Note that this is not a true rollback as in databases. Content you have previously added may have been committed due to autoCommit, buffer full, other client performing a commit etc. It is not a real rollback if you have two threads T1 and T2 that are adding. If T1 is adding 500 and T2 is adding 3 then T2 will commit its 3 document PLUS the documents added by T1 (because T2 will finish add/commit before T2 due to the documents number). Solr transactions are server side only. -- Regards, Amr Ali City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt Ext: 278 -Original Message- From: Jack Krupansky [mailto:jack.krupan...@gmail.com] Sent: Tuesday, May 12, 2015 10:24 PM To: solr-user@lucene.apache.org Subject: Re: Transactional Behavior Solr does have a rollback/ command, but it is an expert feature and not so clear how it works in SolrCloud. See: https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+I ndex+Handlers and https://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22 -- Jack Krupansky On Tue, May 12, 2015 at 12:58 PM, Amr Ali amr_...@siliconexpert.com wrote: Hello, I have a business case in which I need to be able for the rollback. When I tried add/commit I was not able to prevent other threads that write to a given Solr core from committing everything. I also tried indexwriter but Solr did not get changes until we restart it. -- Regards, Amr Ali City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt Ext: 278 -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/
Re: Cannot get uf (user fields) to work
Hi Steve, You use qf to limit search on some fields. Thanks, Emir On 14.05.2015 13:49, Steven White wrote: Hi Everyone, I'm trying to utilize uf but it doesn't work. My reading of it per: https://cwiki.apache.org/confluence/display/solr/The+Extended+DisMax+Query+Parser is I can use it to limit into which fields a search is performed. E.g.: If my index has F1, F2, F3, F4, and I set uf to F3, than I should get hits on F3 only. Here is my request handler: requestHandler name=/select_group_a class=solr.SearchHandler lst name=defaults str name=echoParamsexplicit/str int name=rows20/int str name=defTypeedismax/str str name=qfF1 F2 F3 F4/str str name=ufF3/str str name=fltype,id,score/str str name=wtxml/str str name=indenttrue/str /lst /requestHandler I also tried passing uf on the URL and it too didn't work. I'm getting hits on terms that are in other fields. Is my usage of uf wrong? Thanks Steve -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/
re: A Synonym Searching for Phrase?
I have implemented that but it's not open sourced yet. It will be soon. -Chris. From: Ryan Yacyshyn ryan.yacys...@gmail.com Sent: Thursday, May 14, 2015 12:07 PM To: solr-user@lucene.apache.org Subject: A Synonym Searching for Phrase? Hi All, I'm running into an issue where I have some tokens that really mean the same thing as two. For example, there are a couple ways users might want to search for certain type of visa called the s pass, but they might query for spass or s-pass. I thought I could add a line in my synonym file to solve this, such as: s-pass, spass = s pass This doesn't seem to work. I found an Auto Phrase TokenFilter ( https://github.com/LucidWorks/auto-phrase-tokenfilter) that looks like it might help, but it sounds like it needs to use a specific query parser as well (we're using edismax). Has anyone came across this specific problem before? Would really appreciate your suggestions / help. We're using Solr 4.8.x (and lucidWorks 2.9). Thanks! Ryan
Re: Real-Time get and Dynamic Fields: possible bug.
On Thu, May 14, 2015 at 12:49 PM, Luis Cappa Banda luisca...@gmail.com wrote: If you don' t mark as stored a field indexed and 'facetable', I was expecting to not be able to return their values, so faceting has no sense. Faceting does not use or retrieve stored field values. The labels faceting returns are from the indexed values. If you want the value returned, it needs to be stored only applies to fields in the main document list (the fields that are retrieved for the top ranked documents). -Yonik
Re: Real-Time get and Dynamic Fields: possible bug.
Yep, but those dynamic fields had a field type string, so the unique indexed therm will be the entire field value and the faceted terms counted will match with exactly with each field value. Thats why I was confused. Typically I use faceting with string non tokenized field values for simple stats and this kind of things. Do you think the behavior explained (I mean, ghost dynamic field values when using real-time request handler) can be a bug? I don' t mind investigating it this weekend and trying to patch it. 2015-05-14 18:59 GMT+02:00 Yonik Seeley ysee...@gmail.com: On Thu, May 14, 2015 at 12:49 PM, Luis Cappa Banda luisca...@gmail.com wrote: If you don' t mark as stored a field indexed and 'facetable', I was expecting to not be able to return their values, so faceting has no sense. Faceting does not use or retrieve stored field values. The labels faceting returns are from the indexed values. If you want the value returned, it needs to be stored only applies to fields in the main document list (the fields that are retrieved for the top ranked documents). -Yonik -- - Luis Cappa
Re: Searcher is opening twice on Reload
I suspect you aren't doing anything wrong, i think it's the same as this
bug...
https://issues.apache.org/jira/browse/SOLR-7035
: Date: Thu, 14 May 2015 12:53:34 +0530
: From: Aman Tandon amantandon...@gmail.com
: Reply-To: solr-user@lucene.apache.org
: To: solr-user@lucene.apache.org solr-user@lucene.apache.org
: Subject: Searcher is opening twice on Reload
:
: Hi,
:
: Please help me here, when I am doing the reload of core, my searcher is
: being opening twice. I am also attaching the logs output, please suggest me
: what wrong I am doing here or this is default behavior on reload.
:
: May 14, 2015 12:47:38 PM org.apache.solr.spelling.DirectSolrSpellChecker
: INFO: init:
:
{name=default,field=titlews,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=1,minPrefix=1,maxInspections=5,minQueryLength=5,maxQueryFrequency=100.0,thresholdTokenFrequency=100.0}
: May 14, 2015 12:47:38 PM
: org.apache.solr.handler.component.SpellCheckComponent
: INFO: No queryConverter defined, using default converter
: May 14, 2015 12:47:38 PM
: org.apache.solr.handler.component.QueryElevationComponent
: INFO: Loading QueryElevation from data dir: elevate.xml
: May 14, 2015 12:47:38 PM org.apache.solr.handler.ReplicationHandler
: INFO: Commits will be reserved for 1
: May 14, 2015 12:47:38 PM org.apache.solr.core.QuerySenderListener
: INFO: QuerySenderListener sending requests to Searcher@41dc3c83[IM-Search]
: main{StandardDirectoryReader(segments_dd4:82296:nrt
: _jdq(4.8):C5602938/2310052:delGen=3132
: _jkq(4.8):C6860454/1398005:delGen=2992
: _jx2(4.8):C5237053/1505048:delGen=3241
: _joo(4.8):C5825253/1599671:delGen=3323
: _k4d(4.8):C5860360/1916531:delGen=3150
: _o27(4.8):C5290435/1018865:delGen=370
: _mju(4.8):C5074973/1602707:delGen=1474
: _jka(4.8):C5172599/1774839:delGen=3202
: _nik(4.8):C4698916/1512091:delGen=804 _o8y(4.8):C1137592/521423:delGen=190
: _oeu(4.8):C469094/86291:delGen=29 _odq(4.8):C217505/65596:delGen=55
: _ogd(4.8):C50454/4155:delGen=5 _oea(4.8):C40833/7192:delGen=37
: _ofy(4.8):C73614/7273:delGen=13 _ogx(4.8):C395681/1388:delGen=4
: _ogh(4.8):C7676/70:delGen=2 _ohf(4.8):C108769/21:delGen=2
: _ogc(4.8):C24435/384:delGen=4 _ogi(4.8):C23088/158:delGen=3
: _ogj(4.8):C4217/2:delGen=1 _ohs(4.8):C7 _oh6(4.8):C20509/205:delGen=5
: _oh7(4.8):C3171 _oho(4.8):C6/1:delGen=1 _ohq(4.8):C1
: _ohv(4.8):C10484/996:delGen=2 _ohx(4.8):C500 _ohy(4.8):C1 _ohz(4.8):C1)}
: ^[OFMay 14, 2015 12:47:43 PM org.apache.solr.core.SolrCore
: INFO: [IM-Search] webapp=/solr path=/select
:
params={spellcheck=truelon=0q=qwt=jsonqt=opsview.monitorlat=0rows=0ps=1}
: hits=6 status=0 QTime=1
: May 14, 2015 12:47:44 PM org.apache.solr.core.SolrCore
: INFO: [IM-Search] webapp=null path=null
:
params={start=0event=firstSearcherq=ricedistrib=falseqt=im.search.intentrows=25}
: hits=42749 status=0 QTime=5667
: May 14, 2015 12:47:58 PM org.apache.solr.request.UnInvertedField
: INFO: UnInverted multi-valued field
:
{field=city,memSize=209216385,tindexSize=11029,time=3904,phase1=3783,nTerms=77614,bigTerms=3,termInstances=31291566,uses=0}
: May 14, 2015 12:48:01 PM org.apache.solr.request.UnInvertedField
: INFO: UnInverted multi-valued field
:
{field=biztype,memSize=208847178,tindexSize=40,time=1318,phase1=1193,nTerms=9,bigTerms=4,termInstances=1607459,uses=0}
: May 14, 2015 12:48:01 PM org.apache.solr.core.SolrCore
: INFO: [IM-Search] webapp=null path=null
:
params={start=0event=firstSearcherq=ricedistrib=falseqt=im.searchrows=25}
: hits=57619 status=0 QTime=17194
: May 14, 2015 12:48:04 PM org.apache.solr.core.SolrCore
: INFO: [IM-Search] webapp=null path=null
:
params={start=0event=firstSearcherq=potassium+cyanidedistrib=falseqt=eto.search.offerrows=20}
: hits=443 status=0 QTime=3272
: May 14, 2015 12:48:09 PM org.apache.solr.core.SolrCore
: INFO: [IM-Search] webapp=null path=null
:
params={start=0event=firstSearcherq=motor+spare+partsdistrib=falseqt=im.searchfq=attribs:(locprefglobal+locprefnational+locprefcity)rows=20}
: hits=107297 status=0 QTime=5254
: May 14, 2015 12:48:09 PM org.apache.solr.core.QuerySenderListener
: INFO: QuerySenderListener done.
: May 14, 2015 12:48:09 PM
: org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener
: INFO: Loading spell index for spellchecker: default
: May 14, 2015 12:48:09 PM
: org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener
: INFO: Loading spell index for spellchecker: wordbreak
: May 14, 2015 12:48:09 PM org.apache.solr.core.SolrCore
: INFO: [IM-Search] Registered new searcher Searcher@41dc3c83[IM-Search]
: main{StandardDirectoryReader(segments_dd4:82296:nrt
: _jdq(4.8):C5602938/2310052:delGen=3132
: _jkq(4.8):C6860454/1398005:delGen=2992
: _jx2(4.8):C5237053/1505048:delGen=3241
: _joo(4.8):C5825253/1599671:delGen=3323
: _k4d(4.8):C5860360/1916531:delGen=3150
: _o27(4.8):C5290435/1018865:delGen=370
:
Re: A Synonym Searching for Phrase?
There was a similar scission few days ago, take a look here : I found this very interesting article that I think can help in better understanding the problem : http://lucidworks.com/blog/solution-for-multi-term-synonyms-in-lucenesolr-using-the-auto-phrasing-tokenfilter/ And this : http://opensourceconnections.com/blog/2013/10/27/why-is-multi-term-synonyms-so-hard-in-solr/ Take a look and let me know ! 2015-05-14 17:05 GMT+01:00 Ryan Yacyshyn ryan.yacys...@gmail.com: Hi All, I'm running into an issue where I have some tokens that really mean the same thing as two. For example, there are a couple ways users might want to search for certain type of visa called the s pass, but they might query for spass or s-pass. I thought I could add a line in my synonym file to solve this, such as: s-pass, spass = s pass This doesn't seem to work. I found an Auto Phrase TokenFilter ( https://github.com/LucidWorks/auto-phrase-tokenfilter) that looks like it might help, but it sounds like it needs to use a specific query parser as well (we're using edismax). Has anyone came across this specific problem before? Would really appreciate your suggestions / help. We're using Solr 4.8.x (and lucidWorks 2.9). Thanks! Ryan -- -- Benedetti Alessandro Visiting card : http://about.me/alessandro_benedetti Tyger, tyger burning bright In the forests of the night, What immortal hand or eye Could frame thy fearful symmetry? William Blake - Songs of Experience -1794 England
Re: Searcher is opening twice on Reload
Please help.
The solr version is 4.8.1
With Regards
Aman Tandon
On Thu, May 14, 2015 at 12:53 PM, Aman Tandon amantandon...@gmail.com
wrote:
Hi,
Please help me here, when I am doing the reload of core, my searcher is
being opening twice. I am also attaching the logs output, please suggest me
what wrong I am doing here or this is default behavior on reload.
May 14, 2015 12:47:38 PM org.apache.solr.spelling.DirectSolrSpellChecker
INFO: init:
{name=default,field=titlews,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=1,minPrefix=1,maxInspections=5,minQueryLength=5,maxQueryFrequency=100.0,thresholdTokenFrequency=100.0}
May 14, 2015 12:47:38 PM
org.apache.solr.handler.component.SpellCheckComponent
INFO: No queryConverter defined, using default converter
May 14, 2015 12:47:38 PM
org.apache.solr.handler.component.QueryElevationComponent
INFO: Loading QueryElevation from data dir: elevate.xml
May 14, 2015 12:47:38 PM org.apache.solr.handler.ReplicationHandler
INFO: Commits will be reserved for 1
May 14, 2015 12:47:38 PM org.apache.solr.core.QuerySenderListener
INFO: QuerySenderListener sending requests to Searcher@41dc3c83[IM-Search]
main{StandardDirectoryReader(segments_dd4:82296:nrt
_jdq(4.8):C5602938/2310052:delGen=3132
_jkq(4.8):C6860454/1398005:delGen=2992
_jx2(4.8):C5237053/1505048:delGen=3241
_joo(4.8):C5825253/1599671:delGen=3323
_k4d(4.8):C5860360/1916531:delGen=3150
_o27(4.8):C5290435/1018865:delGen=370
_mju(4.8):C5074973/1602707:delGen=1474
_jka(4.8):C5172599/1774839:delGen=3202
_nik(4.8):C4698916/1512091:delGen=804 _o8y(4.8):C1137592/521423:delGen=190
_oeu(4.8):C469094/86291:delGen=29 _odq(4.8):C217505/65596:delGen=55
_ogd(4.8):C50454/4155:delGen=5 _oea(4.8):C40833/7192:delGen=37
_ofy(4.8):C73614/7273:delGen=13 _ogx(4.8):C395681/1388:delGen=4
_ogh(4.8):C7676/70:delGen=2 _ohf(4.8):C108769/21:delGen=2
_ogc(4.8):C24435/384:delGen=4 _ogi(4.8):C23088/158:delGen=3
_ogj(4.8):C4217/2:delGen=1 _ohs(4.8):C7 _oh6(4.8):C20509/205:delGen=5
_oh7(4.8):C3171 _oho(4.8):C6/1:delGen=1 _ohq(4.8):C1
_ohv(4.8):C10484/996:delGen=2 _ohx(4.8):C500 _ohy(4.8):C1 _ohz(4.8):C1)}
^[OFMay 14, 2015 12:47:43 PM org.apache.solr.core.SolrCore
INFO: [IM-Search] webapp=/solr path=/select
params={spellcheck=truelon=0q=qwt=jsonqt=opsview.monitorlat=0rows=0ps=1}
hits=6 status=0 QTime=1
May 14, 2015 12:47:44 PM org.apache.solr.core.SolrCore
INFO: [IM-Search] webapp=null path=null
params={start=0event=firstSearcherq=ricedistrib=falseqt=im.search.intentrows=25}
hits=42749 status=0 QTime=5667
May 14, 2015 12:47:58 PM org.apache.solr.request.UnInvertedField
INFO: UnInverted multi-valued field
{field=city,memSize=209216385,tindexSize=11029,time=3904,phase1=3783,nTerms=77614,bigTerms=3,termInstances=31291566,uses=0}
May 14, 2015 12:48:01 PM org.apache.solr.request.UnInvertedField
INFO: UnInverted multi-valued field
{field=biztype,memSize=208847178,tindexSize=40,time=1318,phase1=1193,nTerms=9,bigTerms=4,termInstances=1607459,uses=0}
May 14, 2015 12:48:01 PM org.apache.solr.core.SolrCore
INFO: [IM-Search] webapp=null path=null
params={start=0event=firstSearcherq=ricedistrib=falseqt=im.searchrows=25}
hits=57619 status=0 QTime=17194
May 14, 2015 12:48:04 PM org.apache.solr.core.SolrCore
INFO: [IM-Search] webapp=null path=null
params={start=0event=firstSearcherq=potassium+cyanidedistrib=falseqt=eto.search.offerrows=20}
hits=443 status=0 QTime=3272
May 14, 2015 12:48:09 PM org.apache.solr.core.SolrCore
INFO: [IM-Search] webapp=null path=null
params={start=0event=firstSearcherq=motor+spare+partsdistrib=falseqt=im.searchfq=attribs:(locprefglobal+locprefnational+locprefcity)rows=20}
hits=107297 status=0 QTime=5254
May 14, 2015 12:48:09 PM org.apache.solr.core.QuerySenderListener
INFO: QuerySenderListener done.
May 14, 2015 12:48:09 PM
org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener
INFO: Loading spell index for spellchecker: default
May 14, 2015 12:48:09 PM
org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener
INFO: Loading spell index for spellchecker: wordbreak
May 14, 2015 12:48:09 PM org.apache.solr.core.SolrCore
INFO: [IM-Search] Registered new searcher Searcher@41dc3c83[IM-Search]
main{StandardDirectoryReader(segments_dd4:82296:nrt
_jdq(4.8):C5602938/2310052:delGen=3132
_jkq(4.8):C6860454/1398005:delGen=2992
_jx2(4.8):C5237053/1505048:delGen=3241
_joo(4.8):C5825253/1599671:delGen=3323
_k4d(4.8):C5860360/1916531:delGen=3150
_o27(4.8):C5290435/1018865:delGen=370
_mju(4.8):C5074973/1602707:delGen=1474
_jka(4.8):C5172599/1774839:delGen=3202
_nik(4.8):C4698916/1512091:delGen=804 _o8y(4.8):C1137592/521423:delGen=190
_oeu(4.8):C469094/86291:delGen=29 _odq(4.8):C217505/65596:delGen=55
_ogd(4.8):C50454/4155:delGen=5 _oea(4.8):C40833/7192:delGen=37
_ofy(4.8):C73614/7273:delGen=13 _ogx(4.8):C395681/1388:delGen=4
_ogh(4.8):C7676/70:delGen=2 _ohf(4.8):C108769/21:delGen=2
High fieldNorm values causing really odd results
Hi everyone:
From the last couple of week I'm noting some really odd results in my Solr
server, searching for the root cause the one thing I can point out is a very
high value of the fieldNorm parameter in the score calculation, an snippet of
the debug info:
{
match:true,
value:4907799.5,
description:fieldWeight in 5316, product of:,
details:[
{
match:true,
value:2,
description:tf(freq=4.0), with freq of:,
details:[
{
match:true,
value:4,
description:termFreq=4.0
}
]
},
{
match:true,
value:3.7443538,
description:idf(docFreq=49449, maxDocs=769173)
},
{
match:true,
value:655360,
description:fieldNorm(doc=5316)
}
]
}
This match is in the title field, which has 119669 total terms (which isn't
such big number) and the total document count in this index is also rather
small: 613772.
From [1] the fieldNorm encapsulates several parameters in its calculation:
- Field boost (not used by us)
- lengthNorm (which I'm suspecting is the culprit)
I'm thinking on using omitNorms, but first I would like to discover why is this
happening.
Any help on this issue?
Regards,
Re: Real-Time get and Dynamic Fields: possible bug.
That is something I didin' t know, but I thought it was mandatory. I' ll try to explain step by step my (I think) logical way to understand it: - If a field is indexed, you can search by it. - When faceting, you have to index the field (because it can be tokenized and then you would like to facet by their terms). Then, you need to mark as indexed those fields you want to facet by. - If you mark as stored a field, you can return its value with the 'original value' it was stored. - If you facet, you are searching, counting terms and returning values and their counters. Thus, that returning their values step is what I thought where 'stored=true' was necessary. If you don' t mark as stored a field indexed and 'facetable', I was expecting to not be able to return their values, so faceting has no sense. Thats what I thought, of course. If it is not necessary, thats perfect: the lighter the data, the better, and one more thing I' ve learned, :-) Anyway, I think that the question is still open: both are dynamic fields, stored (it is not necessary, OK) and indexed. When applying real time requestHandler, i18n* dynamic fields are returned but those *_facet are not. However, when applying the default /select requestHandler and finding by the document id, both i18n* and *_facet fields are returned. You can try it with Solr 5.1, the version I' m currently using. The only differences between them are: - Regular expression: i18n* VS *_facet - Multivalued: *_facet are multivalued. Regards, - Luis Cappa 2015-05-14 18:32 GMT+02:00 Yonik Seeley ysee...@gmail.com: On Thu, May 14, 2015 at 10:47 AM, Luis Cappa Banda luisca...@gmail.com wrote: Hi Yonik, Yes, they are the target from copyFields in the schema.xml. This *_target fields are suposed to be used in some specific searchable (thus, tokenized) fields that in the future are candidates to be faceted to return some stats. For example, imagine that you have a field storing a directory path and you want to search by. Also, you may want to facet by the whole directory path value (not just their terms). Thats why I' m storing both field values: searchable and tokenized one, string and 'facet candidate' one. OK, but you don't need to *store* the values in _facet, right? -Yonik -- - Luis Cappa
Re: Real-Time get and Dynamic Fields: possible bug.
On Thu, May 14, 2015 at 10:47 AM, Luis Cappa Banda luisca...@gmail.com wrote: Hi Yonik, Yes, they are the target from copyFields in the schema.xml. This *_target fields are suposed to be used in some specific searchable (thus, tokenized) fields that in the future are candidates to be faceted to return some stats. For example, imagine that you have a field storing a directory path and you want to search by. Also, you may want to facet by the whole directory path value (not just their terms). Thats why I' m storing both field values: searchable and tokenized one, string and 'facet candidate' one. OK, but you don't need to *store* the values in _facet, right? -Yonik
A Synonym Searching for Phrase?
Hi All, I'm running into an issue where I have some tokens that really mean the same thing as two. For example, there are a couple ways users might want to search for certain type of visa called the s pass, but they might query for spass or s-pass. I thought I could add a line in my synonym file to solve this, such as: s-pass, spass = s pass This doesn't seem to work. I found an Auto Phrase TokenFilter ( https://github.com/LucidWorks/auto-phrase-tokenfilter) that looks like it might help, but it sounds like it needs to use a specific query parser as well (we're using edismax). Has anyone came across this specific problem before? Would really appreciate your suggestions / help. We're using Solr 4.8.x (and lucidWorks 2.9). Thanks! Ryan
Re: Cannot get uf (user fields) to work
Thanks Shawn and all for the insight. This was very helpful. Steve On Thu, May 14, 2015 at 10:19 AM, Shawn Heisey apa...@elyograg.org wrote: On 5/14/2015 6:33 AM, Steven White wrote: I don't follow. Can you give me an example on how and when to use uf? The uf parameter controls which fields are allowed in the query string. If you have uf=f1 f2 then q=f3:foo won't work the way you would expect. If f3 is in the qf parameter, then it will still be used by the parser. I tried the equivalent of q=f3:foo on my own index with uf set to fields that did not include that field. When a field is disallowed, but used in the query, there is no error message. Instead, the data is interpreted as raw input, so that example would work like q=f3\:foo instead. The colon becomes part of the actual query text. Testing shows that uf does *NOT* affect filter queries, only the main query. I think this is because filter queries default to the lucene parser, not the parser specified in defType. Thanks, Shawn
Re: Suggester
Could you please explain how you solved the problem I am having the same issue :( -- View this message in context: http://lucene.472066.n3.nabble.com/Suggester-tp4201287p4205460.html Sent from the Solr - User mailing list archive at Nabble.com.
NPE when Faceting with MoreLikeThis handler in Solr 5.1.0
Hi everyone, Recently I upgraded to solr 5.1.0. When trying to generate facets using the more like this handler, I now get a a NullPointerException. I never got this exception while using Solr 4.10.0 Details are below: Stack Trace: at org.apache.solr.request.SimpleFacets.getHeatmapCounts(SimpleFacets.java:1555) at org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:284) at org.apache.solr.handler.MoreLikeThisHandler.handleRequestBody(MoreLikeThisHandler.java:233) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:829) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:446) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:368) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) Query: qt=/mlt q=id:545dbb57b54c2403f286050e546dcdcab54cf2d074e5a2f7 mlt.mindf=5 mlt.mintf=1 mlt.minwl=3 mlt.boost=true fq=storeid:546dcdcab54cf2d074e5a2f7 mlt.fl=overview_mlt,abstract_mlt,description_mlt,company_profile_mlt,bio_mlt mlt.interestingTerms=details fl=conceptid,score sort=score desc start=0 rows=2 facet=true facet.field=tags facet.field=locations facet.mincount=1 facet.method=enum facet.limit=-1 facet.sort=count Schema.xml(relevant parts): field name=tags type=string indexed=true stored=true multiValued=true / field name=locations type=string indexed=true stored=true multiValued=true / dynamicField name=*_mlt stored=true indexed=true type=text_general termVectors=true multiValued=true / solrconfig.xml(relevant parts): requestHandler name=/mlt class=solr.MoreLikeThisHandler /requestHandler
Re: High fieldNorm values causing really odd results
: {
: match:true,
: value:655360,
: description:fieldNorm(doc=5316)
: }
...
: This match is in the title field, which has 119669 total terms (which
: isn't such big number) and the total document count in this index is
that smells like a bug -- by the looks of it an overflow bug?
can you please provide some details on the version of solr you are using,
and the specifics of your schema: what field type, what similarity
configuration you have (if any) etc...
-Hoss
http://www.lucidworks.com/
Re: Is copyField a must?
Anyone knows the answer to Shawn's question? Does Solr support POST request and is the format the same as GET? If it does than it means I don't have to create multiple request handlers. Thanks Steve On Wed, May 13, 2015 at 6:02 PM, Shawn Heisey apa...@elyograg.org wrote: On 5/13/2015 3:36 PM, Steven White wrote: Note, I want to avoid a URL base solution (sending the list of fields over HTTP) because the list of fields could be large (1000+) and thus I will exceed GET limit quickly (does Solr support POST for searching, if so, than I can use URL base solution?) Solr does indeed support a query sent as the body in a POST request. I'm not completely positive, but I think you'd use the same format as you put on the URL: q=foorows=1fq=bar If anyone knows for sure what should be in the POST body, please let me and Steven know. In particular, should the content be URL escaped, as might be required for a GET? Thanks, Shawn
RE: Is copyField a must?
Yes, it does support POST. As to format, I believe that's handled by the container. So if you're url-encoding the parameter values, you'll probably need to set Content-Type: application/x-www-form-urlencoded for the HTTP POST header. -Original Message- From: Steven White [mailto:swhite4...@gmail.com] Sent: Thursday, May 14, 2015 3:12 PM To: solr-user@lucene.apache.org Subject: Re: Is copyField a must? Anyone knows the answer to Shawn's question? Does Solr support POST request and is the format the same as GET? If it does than it means I don't have to create multiple request handlers. Thanks Steve On Wed, May 13, 2015 at 6:02 PM, Shawn Heisey apa...@elyograg.org wrote: On 5/13/2015 3:36 PM, Steven White wrote: Note, I want to avoid a URL base solution (sending the list of fields over HTTP) because the list of fields could be large (1000+) and thus I will exceed GET limit quickly (does Solr support POST for searching, if so, than I can use URL base solution?) Solr does indeed support a query sent as the body in a POST request. I'm not completely positive, but I think you'd use the same format as you put on the URL: q=foorows=1fq=bar If anyone knows for sure what should be in the POST body, please let me and Steven know. In particular, should the content be URL escaped, as might be required for a GET? Thanks, Shawn
Re: [MASSMAIL]Re: High fieldNorm values causing really odd results
Hi Hoss,
First of all, thank you for your reply.
Sorry for leaving the Solr version out in my previous email, I'm using Solr
4.10.3 running on Centos7, with the following JRE: Oracle Corporation OpenJDK
64-Bit Server VM (1.7.0_75 24.75-b04)
This are the relevant portions of my schema.xml
!-- Generic text field type --
fieldType name=text class=solr.TextField sortMissingLast=true
analyzer
charFilter class=solr.HTMLStripCharFilterFactory/
tokenizer class=solr.StandardTokenizerFactory/
filter class=solr.ASCIIFoldingFilterFactory/
filter class=solr.StopFilterFactory
ignoreCase=true words=stopwords.txt/
filter class=solr.LowerCaseFilterFactory/
filter class=solr.RemoveDuplicatesTokenFilterFactory/
/analyzer
/fieldType
field name=title type=text stored=true indexed=true
multiValued=true/
In this particular case I'm not using any special features, just a typical text
field. I'm using the default similarity class provided by Solr, this is a
pretty straightforward setup :)
Regards,
- Original Message -
From: Chris Hostetter hossman_luc...@fucit.org
To: solr-user@lucene.apache.org
Sent: Thursday, May 14, 2015 4:08:36 PM
Subject: [MASSMAIL]Re: High fieldNorm values causing really odd results
: {
: match:true,
: value:655360,
: description:fieldNorm(doc=5316)
: }
...
: This match is in the title field, which has 119669 total terms (which
: isn't such big number) and the total document count in this index is
that smells like a bug -- by the looks of it an overflow bug?
can you please provide some details on the version of solr you are using,
and the specifics of your schema: what field type, what similarity
configuration you have (if any) etc...
-Hoss
http://www.lucidworks.com/
Re: [MASSMAIL]Re: High fieldNorm values causing really odd results
: Sorry for leaving the Solr version out in my previous email, I'm using
: Solr 4.10.3 running on Centos7, with the following JRE: Oracle
: Corporation OpenJDK 64-Bit Server VM (1.7.0_75 24.75-b04)
I can't reproduce Using Solr 4.10.3 (or 4.10.4 - mistread your email the
first time)
Are you certain you didn't *build* this index with a different Similarity
configured? or did you perhaps build it with an older version of Solr that
might have had a bug in it?
Here's what i tried...
applied this patch to the example configs based on the fieldType you
specified...
hossman@tray:~/lucene/lucene_solr_4_10_3_tag$ svn diff
Index: solr/example/solr/collection1/conf/schema.xml
===
--- solr/example/solr/collection1/conf/schema.xml (revision 1679472)
+++ solr/example/solr/collection1/conf/schema.xml (working copy)
@@ -46,6 +46,21 @@
--
schema name=example version=1.5
+
+fieldType name=hoss_type class=solr.TextField
sortMissingLast=true
+analyzer
+charFilter class=solr.HTMLStripCharFilterFactory/
+tokenizer class=solr.StandardTokenizerFactory/
+filter class=solr.ASCIIFoldingFilterFactory/
+filter class=solr.StopFilterFactory
+ignoreCase=true words=stopwords.txt/
+filter class=solr.LowerCaseFilterFactory/
+filter class=solr.RemoveDuplicatesTokenFilterFactory/
+/analyzer
+/fieldType
+
+field name=hoss_test type=hoss_type stored=true indexed=true
multiValued=true/
+
!-- attribute name is the name of this schema and is only used for
display purposes.
version=x.y is Solr's version number for the schema syntax and
semantics. It should not normally be changed by applications.
...started up java -jar start.jar and then wrote ran this script to
generate a doc with the number of unique terms in my field that you mentioned
indexed it...
hossman@tray:~/tmp$ cat make-big-field.pl
#/usr/bin/perl
print qq{adddocfield name=idhoss/fieldfield
name=hoss_test\n};
for (1..119669) {
print term${_} ;
}
print qq{/field/doc/add\n};
hossman@tray:~/tmp$ perl make-big-field.pl tmp.xml
hossman@tray:~/tmp$ curl -X POST -H 'Content-Type: application/xml'
--data-binary @tmp.xml
http://localhost:8983/solr/collection1/update?commit=true;
?xml version=1.0 encoding=UTF-8?
response
lst name=responseHeaderint name=status0/intint
name=QTime877/int/lst
/response
Then confirmed i got a very small fieldNorm when querying against this
field...
hossman@tray:~/tmp$ curl
'http://localhost:8983/solr/collection1/select?q=hoss_test:term1debug=resultswt=jsonindent=truefl=idomitHeader=true'
{
response:{numFound:1,start:0,docs:[
{
id:hoss}]
},
debug:{
explain:{
hoss:\n7.491524E-4 = (MATCH) weight(hoss_test:term1 in 0)
[DefaultSimilarity], result of:\n 7.491524E-4 = fieldWeight in 0, product
of:\n1.0 = tf(freq=1.0), with freq of:\n 1.0 = termFreq=1.0\n
0.30685282 = idf(docFreq=1, maxDocs=1)\n0.0024414062 =
fieldNorm(doc=0)\n}}}
-Hoss
http://www.lucidworks.com/
Re: Suggester
This might help: http://lucidworks.com/blog/solr-suggester/ Best, Erick On Thu, May 14, 2015 at 12:43 PM, lreg lise.regn...@gmail.com wrote: Could you please explain how you solved the problem I am having the same issue :( -- View this message in context: http://lucene.472066.n3.nabble.com/Suggester-tp4201287p4205460.html Sent from the Solr - User mailing list archive at Nabble.com.
Full Copy during cluster restarts
We are running a solr 4.10.2 cluster with dozens of nodes. When ever we restart the cluster we see that 2-3 nodes goes in to a fullCopy mode, which takes for ever to recover. Because of the recovery our entire cluster runs very slow and eventually becomes unresponsive. Any thoughts? Thanks Paddy -- Did you breathe today? Learn from here: http://us.artofliving.org/index.html
Real-Time get and Dynamic Fields: possible bug.
Hi there,
I have the following dynamicFields definition in my schema.xml:
!-- I18n DynamicFields --
dynamicField name=i18n* type=string indexed=true stored=true / !--
DynamicFields used typically for faceting issues by copying values from
other existing fields-- dynamicField name=*_facet type=string indexed=
true stored=true multiValued=true /
I' ve seen that when fetching documents with /select?q=id:whateverId, the
results returned include both i18n* and *_facet fields filled. However,
when using real-time request handler (/get?ids:whateverIds) the result
fetched include only i18n* dynamic fields, but *_facet ones are not
included.
I have the impression during /get RequestHandler the server-side regular
expression used when parsing fields and fields values to return documents
with existing dynamic fields seems to be wrong. From the client side, I' ve
checked that the class DocField.java that parses SolrDocument to Bean ones
uses the following matcher:
} else if (annotation.value().indexOf('*') = 0) { // dynamic fields are
annotated as @Field(categories_*)
// if the field was annotated as a dynamic field, convert the name into a
pattern
// the wildcard (*) is supposed to be either a prefix or a suffix, hence
the use of replaceFirst
name = annotation.value().replaceFirst(\\*, \\.*);
dynamicFieldNamePatternMatcher = Pattern.compile(^ + name + $);
} else {
name = annotation.value();
}
So maybe a similar behavior from the server-side is wrong. That' s the only
reason I find to understand why when using /select all fields are returned
but when using /get those that matches *_facet regexp are not.
If you can confirm that this is a bug (because maybe is the expected
behavior, but after some years using Solr I think it is not) I can create
the JIRA issue and debug it more deeply to apply a patch with the aim to
help.
Regards,
--
- Luis Cappa
Re: Wiki new user
You have to create it first, then let us know. Create your account here: https://wiki.apache.org/solr https://wiki.apache.org/solr, from the login page. The let us know the username you create. On May 14, 2015, at 2:49 AM, Sergio Velasco ser...@mitula.com wrote: Hi Erik, SergioVelasco is a good one? If not, SergioMitula or womething like this. Thank you, guy! www.mitula.com http://www.mitula.com/ Sergio Velasco | Dpto. de Desarrollo Contáctame: ser...@mitula.com mailto:ser...@mitula.com | Tfno. +34 917 08 21 47 | Fax +34 917 08 21 56 Síguenos en: Facebook.com/mitula.es.latam http://www.facebook.com/mitula.es.latam | @mitula_es http://twitter.com/mitula_es | Linkedin.com/mitula http://www.linkedin.com/company/mitula.com | Blog http://blog-es.mitula.com/ El contenido de este correo electrónico puede ser confidencial o privilegiado. Si ha recibido este mensaje por error, por favor, no lo reenvíe a nadie. Le rogamos que borre todas las copias, mensajes adjuntos y por favor comuníquenos que lo ha recibido la persona equivocada. Gracias Antes de imprimir este mensaje, asegúrese de que es necesario. El medio ambiente está en nuestra mano De: Erik Hatcher [mailto:erik.hatc...@gmail.com mailto:erik.hatc...@gmail.com] Enviado el: miércoles, 13 de mayo de 2015 16:44 Para: solr-user@lucene.apache.org mailto:solr-user@lucene.apache.org Asunto: Re: Wiki new user Sergio - what is your wiki username? We can add you as an editor once you provide the username. Erik On May 13, 2015, at 10:33, Sergio Velasco ser...@mitula.com mailto:ser...@mitula.com wrote: Hi, I would like to become a member of the Solr wiki. I have requested it to the Solr user lists and they have send me to this list to request access to the wiki. I am the Mitula CTO and we have been using Solr from the very beginning, 6 years ago. I think I can contribute a lot to this wiki. Thank you. www.mitula.com http://www.mitula.com/ Sergio Velasco | Dpto. de Desarrollo Contáctame: ser...@mitula.com mailto:ser...@mitula.com | Tfno. +34 917 08 21 47 | Fax +34 917 08 21 56 Síguenos en: Facebook.com/mitula.es.latam http://www.facebook.com/mitula.es.latam | @mitula_es http://twitter.com/mitula_es | Linkedin.com/mitula http://www.linkedin.com/company/mitula.com | Blog http://blog-es.mitula.com/ El contenido de este correo electrónico puede ser confidencial o privilegiado. Si ha recibido este mensaje por error, por favor, no lo reenvíe a nadie. Le rogamos que borre todas las copias, mensajes adjuntos y por favor comuníquenos que lo ha recibido la persona equivocada. Gracias Antes de imprimir este mensaje, asegúrese de que es necesario. El medio ambiente está en nuestra mano http://www.avast.com/ El software de antivirus Avast ha analizado este correo electrónico en busca de virus. www.avast.com http://www.avast.com/ http://www.avast.com/ El software de antivirus Avast ha analizado este correo electrónico en busca de virus. www.avast.com http://www.avast.com/
Cannot get uf (user fields) to work
Hi Everyone, I'm trying to utilize uf but it doesn't work. My reading of it per: https://cwiki.apache.org/confluence/display/solr/The+Extended+DisMax+Query+Parser is I can use it to limit into which fields a search is performed. E.g.: If my index has F1, F2, F3, F4, and I set uf to F3, than I should get hits on F3 only. Here is my request handler: requestHandler name=/select_group_a class=solr.SearchHandler lst name=defaults str name=echoParamsexplicit/str int name=rows20/int str name=defTypeedismax/str str name=qfF1 F2 F3 F4/str str name=ufF3/str str name=fltype,id,score/str str name=wtxml/str str name=indenttrue/str /lst /requestHandler I also tried passing uf on the URL and it too didn't work. I'm getting hits on terms that are in other fields. Is my usage of uf wrong? Thanks Steve
RE: Wiki new user
Hi Erik, SergioVelasco is a good one? If not, SergioMitula or womething like this. Thank you, guy! http://img.mitula.net/www/mitula/images/firmas/logo_espanol.jpg http://www.mitula.com/ www.mitula.com Sergio Velasco | Dpto. de Desarrollo Contáctame: mailto:ser...@mitula.com ser...@mitula.com | Tfno. +34 917 08 21 47 | Fax +34 917 08 21 56 Síguenos en: http://www.facebook.com/mitula.es.latam Facebook.com/mitula.es.latam | http://twitter.com/mitula_es @mitula_es | http://www.linkedin.com/company/mitula.com Linkedin.com/mitula | http://blog-es.mitula.com/ Blog http://img.mitula.net/www/mitula/images/firmas/mitula-paises.jpg El contenido de este correo electrónico puede ser confidencial o privilegiado. Si ha recibido este mensaje por error, por favor, no lo reenvíe a nadie. Le rogamos que borre todas las copias, mensajes adjuntos y por favor comuníquenos que lo ha recibido la persona equivocada. Gracias Antes de imprimir este mensaje, asegúrese de que es necesario. El medio ambiente está en nuestra mano De: Erik Hatcher [mailto:erik.hatc...@gmail.com] Enviado el: miércoles, 13 de mayo de 2015 16:44 Para: solr-user@lucene.apache.org Asunto: Re: Wiki new user Sergio - what is your wiki username? We can add you as an editor once you provide the username. Erik On May 13, 2015, at 10:33, Sergio Velasco ser...@mitula.com wrote: Hi, I would like to become a member of the Solr wiki. I have requested it to the Solr user lists and they have send me to this list to request access to the wiki. I am the Mitula CTO and we have been using Solr from the very beginning, 6 years ago. I think I can contribute a lot to this wiki. Thank you. http://img.mitula.net/www/mitula/images/firmas/logo_espanol.jpg http://www.mitula.com/ www.mitula.com Sergio Velasco | Dpto. de Desarrollo Contáctame: mailto:ser...@mitula.com ser...@mitula.com | Tfno. +34 917 08 21 47 | Fax +34 917 08 21 56 Síguenos en: http://www.facebook.com/mitula.es.latam Facebook.com/mitula.es.latam | http://twitter.com/mitula_es @mitula_es | http://www.linkedin.com/company/mitula.com Linkedin.com/mitula | http://blog-es.mitula.com/ Blog http://img.mitula.net/www/mitula/images/firmas/mitula-paises.jpg El contenido de este correo electrónico puede ser confidencial o privilegiado. Si ha recibido este mensaje por error, por favor, no lo reenvíe a nadie. Le rogamos que borre todas las copias, mensajes adjuntos y por favor comuníquenos que lo ha recibido la persona equivocada. Gracias Antes de imprimir este mensaje, asegúrese de que es necesario. El medio ambiente está en nuestra mano _ http://www.avast.com/ Imagen quitada por el remitente. Avast logo El software de antivirus Avast ha analizado este correo electrónico en busca de virus. www.avast.com http://www.avast.com/ --- El software de antivirus Avast ha analizado este correo electrónico en busca de virus. http://www.avast.com
Searcher is opening twice on Reload
Hi,
Please help me here, when I am doing the reload of core, my searcher is
being opening twice. I am also attaching the logs output, please suggest me
what wrong I am doing here or this is default behavior on reload.
May 14, 2015 12:47:38 PM org.apache.solr.spelling.DirectSolrSpellChecker
INFO: init:
{name=default,field=titlews,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=1,minPrefix=1,maxInspections=5,minQueryLength=5,maxQueryFrequency=100.0,thresholdTokenFrequency=100.0}
May 14, 2015 12:47:38 PM
org.apache.solr.handler.component.SpellCheckComponent
INFO: No queryConverter defined, using default converter
May 14, 2015 12:47:38 PM
org.apache.solr.handler.component.QueryElevationComponent
INFO: Loading QueryElevation from data dir: elevate.xml
May 14, 2015 12:47:38 PM org.apache.solr.handler.ReplicationHandler
INFO: Commits will be reserved for 1
May 14, 2015 12:47:38 PM org.apache.solr.core.QuerySenderListener
INFO: QuerySenderListener sending requests to Searcher@41dc3c83[IM-Search]
main{StandardDirectoryReader(segments_dd4:82296:nrt
_jdq(4.8):C5602938/2310052:delGen=3132
_jkq(4.8):C6860454/1398005:delGen=2992
_jx2(4.8):C5237053/1505048:delGen=3241
_joo(4.8):C5825253/1599671:delGen=3323
_k4d(4.8):C5860360/1916531:delGen=3150
_o27(4.8):C5290435/1018865:delGen=370
_mju(4.8):C5074973/1602707:delGen=1474
_jka(4.8):C5172599/1774839:delGen=3202
_nik(4.8):C4698916/1512091:delGen=804 _o8y(4.8):C1137592/521423:delGen=190
_oeu(4.8):C469094/86291:delGen=29 _odq(4.8):C217505/65596:delGen=55
_ogd(4.8):C50454/4155:delGen=5 _oea(4.8):C40833/7192:delGen=37
_ofy(4.8):C73614/7273:delGen=13 _ogx(4.8):C395681/1388:delGen=4
_ogh(4.8):C7676/70:delGen=2 _ohf(4.8):C108769/21:delGen=2
_ogc(4.8):C24435/384:delGen=4 _ogi(4.8):C23088/158:delGen=3
_ogj(4.8):C4217/2:delGen=1 _ohs(4.8):C7 _oh6(4.8):C20509/205:delGen=5
_oh7(4.8):C3171 _oho(4.8):C6/1:delGen=1 _ohq(4.8):C1
_ohv(4.8):C10484/996:delGen=2 _ohx(4.8):C500 _ohy(4.8):C1 _ohz(4.8):C1)}
^[OFMay 14, 2015 12:47:43 PM org.apache.solr.core.SolrCore
INFO: [IM-Search] webapp=/solr path=/select
params={spellcheck=truelon=0q=qwt=jsonqt=opsview.monitorlat=0rows=0ps=1}
hits=6 status=0 QTime=1
May 14, 2015 12:47:44 PM org.apache.solr.core.SolrCore
INFO: [IM-Search] webapp=null path=null
params={start=0event=firstSearcherq=ricedistrib=falseqt=im.search.intentrows=25}
hits=42749 status=0 QTime=5667
May 14, 2015 12:47:58 PM org.apache.solr.request.UnInvertedField
INFO: UnInverted multi-valued field
{field=city,memSize=209216385,tindexSize=11029,time=3904,phase1=3783,nTerms=77614,bigTerms=3,termInstances=31291566,uses=0}
May 14, 2015 12:48:01 PM org.apache.solr.request.UnInvertedField
INFO: UnInverted multi-valued field
{field=biztype,memSize=208847178,tindexSize=40,time=1318,phase1=1193,nTerms=9,bigTerms=4,termInstances=1607459,uses=0}
May 14, 2015 12:48:01 PM org.apache.solr.core.SolrCore
INFO: [IM-Search] webapp=null path=null
params={start=0event=firstSearcherq=ricedistrib=falseqt=im.searchrows=25}
hits=57619 status=0 QTime=17194
May 14, 2015 12:48:04 PM org.apache.solr.core.SolrCore
INFO: [IM-Search] webapp=null path=null
params={start=0event=firstSearcherq=potassium+cyanidedistrib=falseqt=eto.search.offerrows=20}
hits=443 status=0 QTime=3272
May 14, 2015 12:48:09 PM org.apache.solr.core.SolrCore
INFO: [IM-Search] webapp=null path=null
params={start=0event=firstSearcherq=motor+spare+partsdistrib=falseqt=im.searchfq=attribs:(locprefglobal+locprefnational+locprefcity)rows=20}
hits=107297 status=0 QTime=5254
May 14, 2015 12:48:09 PM org.apache.solr.core.QuerySenderListener
INFO: QuerySenderListener done.
May 14, 2015 12:48:09 PM
org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener
INFO: Loading spell index for spellchecker: default
May 14, 2015 12:48:09 PM
org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener
INFO: Loading spell index for spellchecker: wordbreak
May 14, 2015 12:48:09 PM org.apache.solr.core.SolrCore
INFO: [IM-Search] Registered new searcher Searcher@41dc3c83[IM-Search]
main{StandardDirectoryReader(segments_dd4:82296:nrt
_jdq(4.8):C5602938/2310052:delGen=3132
_jkq(4.8):C6860454/1398005:delGen=2992
_jx2(4.8):C5237053/1505048:delGen=3241
_joo(4.8):C5825253/1599671:delGen=3323
_k4d(4.8):C5860360/1916531:delGen=3150
_o27(4.8):C5290435/1018865:delGen=370
_mju(4.8):C5074973/1602707:delGen=1474
_jka(4.8):C5172599/1774839:delGen=3202
_nik(4.8):C4698916/1512091:delGen=804 _o8y(4.8):C1137592/521423:delGen=190
_oeu(4.8):C469094/86291:delGen=29 _odq(4.8):C217505/65596:delGen=55
_ogd(4.8):C50454/4155:delGen=5 _oea(4.8):C40833/7192:delGen=37
_ofy(4.8):C73614/7273:delGen=13 _ogx(4.8):C395681/1388:delGen=4
_ogh(4.8):C7676/70:delGen=2 _ohf(4.8):C108769/21:delGen=2
_ogc(4.8):C24435/384:delGen=4 _ogi(4.8):C23088/158:delGen=3
_ogj(4.8):C4217/2:delGen=1 _ohs(4.8):C7 _oh6(4.8):C20509/205:delGen=5
_oh7(4.8):C3171
Re: [MASSMAIL]Re: High fieldNorm values causing really odd results
Regarding the experiment, sorry If I explained myself in the wrong way, the
indexed document doesn't have 119669 terms have a lot less terms (less than a
1000 terms, I don't have the exact number here now), instead 119669 is the
number of distinct terms reported by luke (Top-terms total in the admin
interface) on the title field.
This index was built from scratch using 4.10.3 if I'm no remembering
incorrectly. Perhaps part of the data could be indexed using 4.10.2, but we
updated our box quite some time ago and this problem didn't appear until
recently. The more strange issue is that this was working fine until a week or
so ago, the only thing I found strange is that the root partition in our Solr
box got out of space; basically we've Solr deployed in Tomcat, which is
installed in the root partition but the cores and all Solr related data is
stored in a separated partition mounted in /opt with plenty of space to grow;
could this be the cause of this behavior?
We're thinking on rebuilding our index, but would love to avoid it if possible
and more importantly find the root cause if this issue (if is possible at all).
As I said before very grateful for your responses,
- Original Message -
From: Chris Hostetter hossman_luc...@fucit.org
To: solr-user@lucene.apache.org
Sent: Thursday, May 14, 2015 7:11:08 PM
Subject: Re: [MASSMAIL]Re: High fieldNorm values causing really odd results
: Sorry for leaving the Solr version out in my previous email, I'm using
: Solr 4.10.3 running on Centos7, with the following JRE: Oracle
: Corporation OpenJDK 64-Bit Server VM (1.7.0_75 24.75-b04)
I can't reproduce Using Solr 4.10.3 (or 4.10.4 - mistread your email the
first time)
Are you certain you didn't *build* this index with a different Similarity
configured? or did you perhaps build it with an older version of Solr that
might have had a bug in it?
Here's what i tried...
applied this patch to the example configs based on the fieldType you
specified...
hossman@tray:~/lucene/lucene_solr_4_10_3_tag$ svn diff
Index: solr/example/solr/collection1/conf/schema.xml
===
--- solr/example/solr/collection1/conf/schema.xml (revision 1679472)
+++ solr/example/solr/collection1/conf/schema.xml (working copy)
@@ -46,6 +46,21 @@
--
schema name=example version=1.5
+
+fieldType name=hoss_type class=solr.TextField
sortMissingLast=true
+analyzer
+charFilter class=solr.HTMLStripCharFilterFactory/
+tokenizer class=solr.StandardTokenizerFactory/
+filter class=solr.ASCIIFoldingFilterFactory/
+filter class=solr.StopFilterFactory
+ignoreCase=true words=stopwords.txt/
+filter class=solr.LowerCaseFilterFactory/
+filter class=solr.RemoveDuplicatesTokenFilterFactory/
+/analyzer
+/fieldType
+
+field name=hoss_test type=hoss_type stored=true indexed=true
multiValued=true/
+
!-- attribute name is the name of this schema and is only used for
display purposes.
version=x.y is Solr's version number for the schema syntax and
semantics. It should not normally be changed by applications.
...started up java -jar start.jar and then wrote ran this script to
generate a doc with the number of unique terms in my field that you mentioned
indexed it...
hossman@tray:~/tmp$ cat make-big-field.pl
#/usr/bin/perl
print qq{adddocfield name=idhoss/fieldfield
name=hoss_test\n};
for (1..119669) {
print term${_} ;
}
print qq{/field/doc/add\n};
hossman@tray:~/tmp$ perl make-big-field.pl tmp.xml
hossman@tray:~/tmp$ curl -X POST -H 'Content-Type: application/xml'
--data-binary @tmp.xml
http://localhost:8983/solr/collection1/update?commit=true;
?xml version=1.0 encoding=UTF-8?
response
lst name=responseHeaderint name=status0/intint
name=QTime877/int/lst
/response
Then confirmed i got a very small fieldNorm when querying against this
field...
hossman@tray:~/tmp$ curl
'http://localhost:8983/solr/collection1/select?q=hoss_test:term1debug=resultswt=jsonindent=truefl=idomitHeader=true'
{
response:{numFound:1,start:0,docs:[
{
id:hoss}]
},
debug:{
explain:{
hoss:\n7.491524E-4 = (MATCH) weight(hoss_test:term1 in 0)
[DefaultSimilarity], result of:\n 7.491524E-4 = fieldWeight in 0, product
of:\n1.0 = tf(freq=1.0), with freq of:\n 1.0 = termFreq=1.0\n
0.30685282 = idf(docFreq=1, maxDocs=1)\n0.0024414062 =
fieldNorm(doc=0)\n}}}
-Hoss
http://www.lucidworks.com/
