Solr 7.6 Error deleting copy field
Hi All,
I have exact problem mentioned in this link below.
http://lucene.472066.n3.nabble.com/Error-deleting-copy-field-td4393097.html
I want to delete all the copy fields with the same source name
curl http://localhost:8983/solr/pi_squared/schema/copyfields
{
"source":"abstract_txt_en",
"dest":"catchall"},
{
"source":"abstract_txt_en",
"dest":"catchall"},
{
"source":"abstract_txt_en",
"dest":"catchall"},
{
"source":"abstract_txt_en",
"dest":"catchall"},
But when I try to delete with the delete-copy-field, I get a following
error
"errorMessages":["Copy field directive not found: 'abstract_txt_en' ->
'catchall'\n"]}],
abstract_txt_en is a dynamic field
I even tried with "source":"*_txt_en", but I get a same error
curl -X POST -H 'Content-type:application/json' --data-binary '{
"delete-copy-field":{ "source":"*_txt_en", "dest":"catchall" }
}' http://localhost:8983/solr/pi_squared/schema
Thank you for you help in advance
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: indexed and stored for fields that are sources of a copy field
Hi Chris, Even better - you can contribute with documentation - you can create jira with patch. Thanks, Emir -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > On 22 Oct 2018, at 15:43, Chris Wareham > wrote: > > Hi Emir, > > Many thanks for the confirmation. I'd kind of inferred this was correct > from the paragraph starting with "Copying is done at the stream source > level", but it would be good to mention it in the "Copying Fields" > section of the Solr documentation. Should I create a JIRA issue asking > for this? > > Regards, > > Chris > > On 22/10/2018 14:28, Emir Arnautović wrote: >> Hi Chris, >> Yes you can do that. There is also type=“ignored” that you can use in such >> scenario. >> HTH, >> Emir >> -- >> Monitoring - Log Management - Alerting - Anomaly Detection >> Solr & Elasticsearch Consulting Support Training - http://sematext.com/ >>> On 22 Oct 2018, at 15:22, Chris Wareham >>> wrote: >>> >>> Hi folks, >>> >>> I have a number of fields defined in my managed-schema file that are used >>> as the sources for a copy field: >>> >>> >> stored="true"/> >>> >> stored="true" multiValued="true"/> >>> >> stored="true" multiValued="true"/> >>> >>> >> stored="false" multiValued="true"/> >>> >>> >>> >>> >>> >>> Can I set both the indexed and stored values to false for the body, sectors >>> and locations fields since I don't want to search or retrieve them? >>> >>> Regards, >>> >>> Chris
Re: indexed and stored for fields that are sources of a copy field
Hi Emir, Many thanks for the confirmation. I'd kind of inferred this was correct from the paragraph starting with "Copying is done at the stream source level", but it would be good to mention it in the "Copying Fields" section of the Solr documentation. Should I create a JIRA issue asking for this? Regards, Chris On 22/10/2018 14:28, Emir Arnautović wrote: Hi Chris, Yes you can do that. There is also type=“ignored” that you can use in such scenario. HTH, Emir -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/ On 22 Oct 2018, at 15:22, Chris Wareham wrote: Hi folks, I have a number of fields defined in my managed-schema file that are used as the sources for a copy field: Can I set both the indexed and stored values to false for the body, sectors and locations fields since I don't want to search or retrieve them? Regards, Chris
Re: indexed and stored for fields that are sources of a copy field
Hi Chris, Yes you can do that. There is also type=“ignored” that you can use in such scenario. HTH, Emir -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > On 22 Oct 2018, at 15:22, Chris Wareham > wrote: > > Hi folks, > > I have a number of fields defined in my managed-schema file that are used as > the sources for a copy field: > > stored="true"/> > stored="true" multiValued="true"/> > stored="true" multiValued="true"/> > > stored="false" multiValued="true"/> > > > > > > Can I set both the indexed and stored values to false for the body, sectors > and locations fields since I don't want to search or retrieve them? > > Regards, > > Chris
indexed and stored for fields that are sources of a copy field
Hi folks, I have a number of fields defined in my managed-schema file that are used as the sources for a copy field: stored="true"/> stored="true" multiValued="true"/> stored="true" multiValued="true"/> stored="false" multiValued="true"/> Can I set both the indexed and stored values to false for the body, sectors and locations fields since I don't want to search or retrieve them? Regards, Chris
Re: copy field
You are right Eric. But range facet does not have support for limit, offset and sort. Thanks, Anil On 13 July 2018 at 19:32, Erick Erickson wrote: > Consider a range facet where you specify a start date, end date and "gap". > The "date math" bits allow you to specify gaps like +1HOUR, +1DAY etc. > > https://lucene.apache.org/solr/guide/7_4/faceting.html > > Best, > Erick > > On Fri, Jul 13, 2018 at 3:39 AM, Andrea Gazzarini > wrote: > > You're welcome, > > but keep in mind what Erik said: it seems you don't need that information > > at "view" level so maybe with range faceting you could get what you need > > without creating additional (and redundant) fields. > > > > Andrea > > > > Il ven 13 lug 2018, 07:04 Anil ha scritto: > > > >> Thanks Andrea. i will write update processor in index pipe line. > >> > >> I feel this is very good feature to support. > >> > >> Thanks, > >> Anil > >> > >> On 12 July 2018 at 22:59, Andrea Gazzarini > wrote: > >> > >> > Hi Anil, > >> > The copy Field directive is not what you're looking for because it > >> doesn't > >> > change the stored value of a field. > >> > > >> > What you need is an Update Request Processor, which is a kind of > >> > interceptor in the indexing chain (i.e. It allows you to change an > >> incoming > >> > document before it gets indexed). > >> > Unfortunately, as far as I know there's not an available processor for > >> > doing what you need in the example you described, but consider that > >> writing > >> > a new processor is a trivial thing. > >> > > >> > Andrea > >> > > >> > Il gio 12 lug 2018, 19:23 Anil ha scritto: > >> > > >> > > HI, > >> > > > >> > > i have a date field which needs to copied to different field with > >> > different > >> > > format/value. is there any way to achieve this using copy field ? > or > >> > needs > >> > > to be done when creating solr document itself. > >> > > > >> > > lets say createdDate is 10-23-2017 10:15:00, it needs to be copied > to > >> > > transformedDate field as 10-23-2017. > >> > > > >> > > please help. thanks. > >> > > > >> > > Regards, > >> > > Anil > >> > > > >> > > >> >
Re: copy field
Consider a range facet where you specify a start date, end date and "gap". The "date math" bits allow you to specify gaps like +1HOUR, +1DAY etc. https://lucene.apache.org/solr/guide/7_4/faceting.html Best, Erick On Fri, Jul 13, 2018 at 3:39 AM, Andrea Gazzarini wrote: > You're welcome, > but keep in mind what Erik said: it seems you don't need that information > at "view" level so maybe with range faceting you could get what you need > without creating additional (and redundant) fields. > > Andrea > > Il ven 13 lug 2018, 07:04 Anil ha scritto: > >> Thanks Andrea. i will write update processor in index pipe line. >> >> I feel this is very good feature to support. >> >> Thanks, >> Anil >> >> On 12 July 2018 at 22:59, Andrea Gazzarini wrote: >> >> > Hi Anil, >> > The copy Field directive is not what you're looking for because it >> doesn't >> > change the stored value of a field. >> > >> > What you need is an Update Request Processor, which is a kind of >> > interceptor in the indexing chain (i.e. It allows you to change an >> incoming >> > document before it gets indexed). >> > Unfortunately, as far as I know there's not an available processor for >> > doing what you need in the example you described, but consider that >> writing >> > a new processor is a trivial thing. >> > >> > Andrea >> > >> > Il gio 12 lug 2018, 19:23 Anil ha scritto: >> > >> > > HI, >> > > >> > > i have a date field which needs to copied to different field with >> > different >> > > format/value. is there any way to achieve this using copy field ? or >> > needs >> > > to be done when creating solr document itself. >> > > >> > > lets say createdDate is 10-23-2017 10:15:00, it needs to be copied to >> > > transformedDate field as 10-23-2017. >> > > >> > > please help. thanks. >> > > >> > > Regards, >> > > Anil >> > > >> > >>
Re: copy field
You're welcome, but keep in mind what Erik said: it seems you don't need that information at "view" level so maybe with range faceting you could get what you need without creating additional (and redundant) fields. Andrea Il ven 13 lug 2018, 07:04 Anil ha scritto: > Thanks Andrea. i will write update processor in index pipe line. > > I feel this is very good feature to support. > > Thanks, > Anil > > On 12 July 2018 at 22:59, Andrea Gazzarini wrote: > > > Hi Anil, > > The copy Field directive is not what you're looking for because it > doesn't > > change the stored value of a field. > > > > What you need is an Update Request Processor, which is a kind of > > interceptor in the indexing chain (i.e. It allows you to change an > incoming > > document before it gets indexed). > > Unfortunately, as far as I know there's not an available processor for > > doing what you need in the example you described, but consider that > writing > > a new processor is a trivial thing. > > > > Andrea > > > > Il gio 12 lug 2018, 19:23 Anil ha scritto: > > > > > HI, > > > > > > i have a date field which needs to copied to different field with > > different > > > format/value. is there any way to achieve this using copy field ? or > > needs > > > to be done when creating solr document itself. > > > > > > lets say createdDate is 10-23-2017 10:15:00, it needs to be copied to > > > transformedDate field as 10-23-2017. > > > > > > please help. thanks. > > > > > > Regards, > > > Anil > > > > > >
Re: copy field
Thanks Andrea. i will write update processor in index pipe line. I feel this is very good feature to support. Thanks, Anil On 12 July 2018 at 22:59, Andrea Gazzarini wrote: > Hi Anil, > The copy Field directive is not what you're looking for because it doesn't > change the stored value of a field. > > What you need is an Update Request Processor, which is a kind of > interceptor in the indexing chain (i.e. It allows you to change an incoming > document before it gets indexed). > Unfortunately, as far as I know there's not an available processor for > doing what you need in the example you described, but consider that writing > a new processor is a trivial thing. > > Andrea > > Il gio 12 lug 2018, 19:23 Anil ha scritto: > > > HI, > > > > i have a date field which needs to copied to different field with > different > > format/value. is there any way to achieve this using copy field ? or > needs > > to be done when creating solr document itself. > > > > lets say createdDate is 10-23-2017 10:15:00, it needs to be copied to > > transformedDate field as 10-23-2017. > > > > please help. thanks. > > > > Regards, > > Anil > > >
Re: copy field
HI Eric, i have a schema with a date field (tweetedDate) with 10-23-2017 10:15:00 format. Same schema is used with number of collections. i have to index the date field with different formats in different collections. lets says collections as collection-day, collection-hour, etc if the date field( tweetedDate) value is 10-23-2017 10:15:00 date format in collection day - 10-23-2017 *00:00:00* * date format in collection hour - 10-23-2017 10:00:00 * so that i can use that date field for faceting for aggregation. my idea is to create new date field (newTweetedDate) which holds the transformed value which is used for aggregation. One way of indexing newTweetedDate is to perform the transformation based on collection and set to solr document for indexing. I am checking if there is any other alternative way where newTweetedDate is derived from tweetedDate automatically using schema using copyField or some other feature. Hope this is clear. thanks. Regards, Anil On 12 July 2018 at 23:04, Erick Erickson wrote: > This seems like an XY problem, you've asked how to do X without > explaining _why_ (the Y). > > If this is just because you want to search the field without having > to specify the full string, consider a DateRangeField. > > Best, > Erick > > On Thu, Jul 12, 2018 at 10:22 AM, Anil wrote: > > HI, > > > > i have a date field which needs to copied to different field with > different > > format/value. is there any way to achieve this using copy field ? or > needs > > to be done when creating solr document itself. > > > > lets say createdDate is 10-23-2017 10:15:00, it needs to be copied to > > transformedDate field as 10-23-2017. > > > > please help. thanks. > > > > Regards, > > Anil >
Re: copy field
Gus,
Perhaps you might try the technique described in the forwarded exchange
below. It has been working very nicely for me.
Terry
Forwarded Message
Subject:Re: Changing Field Assignments
Date: Tue, 12 Jun 2018 12:21:16 +0900
From: Yasufumi Mizoguchi
Reply-To: solr-user@lucene.apache.org
To: solr-user@lucene.apache.org
Hi,
You can do that via adding the following lines in managed-schema.
After adding the above and re-indexing docs, you will get the result like
following.
{ "responseHeader":{ "status":0, "QTime":0, "params":{ "q":"*:*", "indent":
"on", "wt":"json", "_":"1528772599296"}}, "response":{"numFound":2,"start":0
,"docs":[ { "id":"test2", "meta_creation_date":["2018-04-30T00:00:00Z"], "
meta_creation_date_range":"2018-04-30T00:00:00Z", "_version_":
1603034044781559808}, { "id":"test", "meta_creation_date":[
"1944-04-02T00:00:00Z"], "meta_creation_date_range":"1944-04-02T00:00:00Z",
"_version_":1603034283921899520}] }}
thanks,
Yasufumi
2018年6月12日(火) 5:04 Terry Steichen :
> I am using Solr (6.6.0) in the automatic mode (where it discovers
> fields). It's working fine with one exception. The problem is that
> Solr maps the discovered "meta_creation_date" is assigned the type
> TrieDateField.
>
> Unfortunately, that type is limited in a number of ways (like sorting,
> abbreviated forms and etc.). What I'd like to do is have that
> ("meta_creation_date") field assigned to a different type, like
> DateRangeField.
>
> Is it possible to accomplish this (during indexing) by creating a copy
> field to a different type, and using the copy field in the query? Or
> via some kind of function operation (which I've never understood)?
>
>
On 07/12/2018 02:43 PM, Gus Heck wrote:
> XY question not withstanding, this is exactly the sort of thing one might
> want to do in their indexing pipeline. For example:
>
> https://github.com/nsoft/jesterj/blob/master/code/ingest/src/main/java/org/jesterj/ingest/processors/SimpleDateTimeReformatter.java
>
> On Thu, Jul 12, 2018 at 1:34 PM, Erick Erickson
> wrote:
>
>> This seems like an XY problem, you've asked how to do X without
>> explaining _why_ (the Y).
>>
>> If this is just because you want to search the field without having
>> to specify the full string, consider a DateRangeField.
>>
>> Best,
>> Erick
>>
>> On Thu, Jul 12, 2018 at 10:22 AM, Anil wrote:
>>> HI,
>>>
>>> i have a date field which needs to copied to different field with
>> different
>>> format/value. is there any way to achieve this using copy field ? or
>> needs
>>> to be done when creating solr document itself.
>>>
>>> lets say createdDate is 10-23-2017 10:15:00, it needs to be copied to
>>> transformedDate field as 10-23-2017.
>>>
>>> please help. thanks.
>>>
>>> Regards,
>>> Anil
>
>
Re: copy field
XY question not withstanding, this is exactly the sort of thing one might want to do in their indexing pipeline. For example: https://github.com/nsoft/jesterj/blob/master/code/ingest/src/main/java/org/jesterj/ingest/processors/SimpleDateTimeReformatter.java On Thu, Jul 12, 2018 at 1:34 PM, Erick Erickson wrote: > This seems like an XY problem, you've asked how to do X without > explaining _why_ (the Y). > > If this is just because you want to search the field without having > to specify the full string, consider a DateRangeField. > > Best, > Erick > > On Thu, Jul 12, 2018 at 10:22 AM, Anil wrote: > > HI, > > > > i have a date field which needs to copied to different field with > different > > format/value. is there any way to achieve this using copy field ? or > needs > > to be done when creating solr document itself. > > > > lets say createdDate is 10-23-2017 10:15:00, it needs to be copied to > > transformedDate field as 10-23-2017. > > > > please help. thanks. > > > > Regards, > > Anil > -- http://www.the111shift.com
Re: copy field
This seems like an XY problem, you've asked how to do X without explaining _why_ (the Y). If this is just because you want to search the field without having to specify the full string, consider a DateRangeField. Best, Erick On Thu, Jul 12, 2018 at 10:22 AM, Anil wrote: > HI, > > i have a date field which needs to copied to different field with different > format/value. is there any way to achieve this using copy field ? or needs > to be done when creating solr document itself. > > lets say createdDate is 10-23-2017 10:15:00, it needs to be copied to > transformedDate field as 10-23-2017. > > please help. thanks. > > Regards, > Anil
Re: copy field
Hi Anil, The copy Field directive is not what you're looking for because it doesn't change the stored value of a field. What you need is an Update Request Processor, which is a kind of interceptor in the indexing chain (i.e. It allows you to change an incoming document before it gets indexed). Unfortunately, as far as I know there's not an available processor for doing what you need in the example you described, but consider that writing a new processor is a trivial thing. Andrea Il gio 12 lug 2018, 19:23 Anil ha scritto: > HI, > > i have a date field which needs to copied to different field with different > format/value. is there any way to achieve this using copy field ? or needs > to be done when creating solr document itself. > > lets say createdDate is 10-23-2017 10:15:00, it needs to be copied to > transformedDate field as 10-23-2017. > > please help. thanks. > > Regards, > Anil >
copy field
HI, i have a date field which needs to copied to different field with different format/value. is there any way to achieve this using copy field ? or needs to be done when creating solr document itself. lets say createdDate is 10-23-2017 10:15:00, it needs to be copied to transformedDate field as 10-23-2017. please help. thanks. Regards, Anil
Re: Error deleting copy field
Ikenna, Can you please create a JIRA issue[1], and if possible include your schema? Thanks. -- Steve www.lucidworks.com [1] https://wiki.apache.org/solr/HowToContribute#JIRA_tips_.28our_issue.2Fbug_tracker.29 > On Jun 6, 2018, at 5:16 PM, Ikenna Okonkwo wrote: > > Hi Steve, > > I didn't use a dynamic field. It was an exact match. > > -- > Ikenna Okonkwo > Senior Front End Developer, WMX Project > Phone: (905) 821-2111 (Ext: 78847) > Mobile: (437) 922-9004 > ikenna.okon...@walmart.com > > Walmart Canada Corp. > 1940 Argentia Road > Mississauga, Ontario L5N 1P9 > Save money. Live better. > > On 2018-06-06, 5:07 PM, "Steve Rowe" wrote: > >Hi Ikenna, > >If the source field(s) are dynamic fields, then you have to give the same > spelling you used when you created the copy field. E.g. for ‘item_desc_en’, > if the source field in the copy field directive is ‘*_desc_en’, then you have > to use that when you issue the delete. > >You can find out the exact spelling of your copy field rules by looking at > the schema. > >-- >Steve >www.lucidworks.com > >> On Jun 6, 2018, at 5:00 PM, Ikenna Okonkwo >> wrote: >> >> Hi all, >> >> I encountered an error when using SOLR cloud 7.2.1. I used the UI to add a >> bunch of “copy fields” to a field named “_text_”, and now, I can’t delete >> the copy fields nor the field. >> >> I get the error message when I attempt to delete a copy field: Copy field >> directive not found: 'item_desc_en' -> '_text_' >> And I get the following when I try to delete the field itself: Can't delete >> field '_text_' because it's referred to by at least one copy field directive. >> >> Any pointers would be helpful. >> >> Thanks! >> >> >> -- >> Ikenna Okonkwo >> Senior Front End Developer, WMX Project >> Phone: (905) 821-2111 (Ext: 78847) >> ikenna.okon...@walmart.com<mailto:ikenna.okon...@walmart.com> >> >> Walmart Canada Corp. >> 1940 Argentia Road >> Mississauga, Ontario L5N 1P9 >> Save money. Live better. > > > >
Re: Error deleting copy field
Hi Steve, I didn't use a dynamic field. It was an exact match. -- Ikenna Okonkwo Senior Front End Developer, WMX Project Phone: (905) 821-2111 (Ext: 78847) Mobile: (437) 922-9004 ikenna.okon...@walmart.com Walmart Canada Corp. 1940 Argentia Road Mississauga, Ontario L5N 1P9 Save money. Live better. On 2018-06-06, 5:07 PM, "Steve Rowe" wrote: Hi Ikenna, If the source field(s) are dynamic fields, then you have to give the same spelling you used when you created the copy field. E.g. for ‘item_desc_en’, if the source field in the copy field directive is ‘*_desc_en’, then you have to use that when you issue the delete. You can find out the exact spelling of your copy field rules by looking at the schema. -- Steve www.lucidworks.com > On Jun 6, 2018, at 5:00 PM, Ikenna Okonkwo wrote: > > Hi all, > > I encountered an error when using SOLR cloud 7.2.1. I used the UI to add a bunch of “copy fields” to a field named “_text_”, and now, I can’t delete the copy fields nor the field. > > I get the error message when I attempt to delete a copy field: Copy field directive not found: 'item_desc_en' -> '_text_' > And I get the following when I try to delete the field itself: Can't delete field '_text_' because it's referred to by at least one copy field directive. > > Any pointers would be helpful. > > Thanks! > > > -- > Ikenna Okonkwo > Senior Front End Developer, WMX Project > Phone: (905) 821-2111 (Ext: 78847) > ikenna.okon...@walmart.com<mailto:ikenna.okon...@walmart.com> > > Walmart Canada Corp. > 1940 Argentia Road > Mississauga, Ontario L5N 1P9 > Save money. Live better.
Re: Error deleting copy field
Hi Ikenna, If the source field(s) are dynamic fields, then you have to give the same spelling you used when you created the copy field. E.g. for ‘item_desc_en’, if the source field in the copy field directive is ‘*_desc_en’, then you have to use that when you issue the delete. You can find out the exact spelling of your copy field rules by looking at the schema. -- Steve www.lucidworks.com > On Jun 6, 2018, at 5:00 PM, Ikenna Okonkwo wrote: > > Hi all, > > I encountered an error when using SOLR cloud 7.2.1. I used the UI to add a > bunch of “copy fields” to a field named “_text_”, and now, I can’t delete the > copy fields nor the field. > > I get the error message when I attempt to delete a copy field: Copy field > directive not found: 'item_desc_en' -> '_text_' > And I get the following when I try to delete the field itself: Can't delete > field '_text_' because it's referred to by at least one copy field directive. > > Any pointers would be helpful. > > Thanks! > > > -- > Ikenna Okonkwo > Senior Front End Developer, WMX Project > Phone: (905) 821-2111 (Ext: 78847) > ikenna.okon...@walmart.com<mailto:ikenna.okon...@walmart.com> > > Walmart Canada Corp. > 1940 Argentia Road > Mississauga, Ontario L5N 1P9 > Save money. Live better.
Error deleting copy field
Hi all, I encountered an error when using SOLR cloud 7.2.1. I used the UI to add a bunch of “copy fields” to a field named “_text_”, and now, I can’t delete the copy fields nor the field. I get the error message when I attempt to delete a copy field: Copy field directive not found: 'item_desc_en' -> '_text_' And I get the following when I try to delete the field itself: Can't delete field '_text_' because it's referred to by at least one copy field directive. Any pointers would be helpful. Thanks! -- Ikenna Okonkwo Senior Front End Developer, WMX Project Phone: (905) 821-2111 (Ext: 78847) ikenna.okon...@walmart.com<mailto:ikenna.okon...@walmart.com> Walmart Canada Corp. 1940 Argentia Road Mississauga, Ontario L5N 1P9 Save money. Live better.
Re: Copy field on dynamic fields?
: Have you tried reading existing example schemas? They show various : permutations of copy fields. Hmm... as the example schema's have been simplified/consolidated/purged it seems we have lost the specific examples that are relevant to the users question -- the only instance of a glob'ed copyField in any of the configsets we ship is with a single destination field. And the ref guide doesn't mention globs in copyField dest either? (created SOLR-12191) Jatin: what you are asking about is 100% possible -- here's some examples from one of our test configs used specifically for testing copyField... This ensures that any field name starting with "dynamic_" is also copied to an "equivilent" field name *ending* with "_dynamic" so "1234_dynamic" gets copied to "dynamic_1234", "foo_dynamic" gets copied to "dynamic_foo" etc... This "glob" pattern in copyFields also works even if the underlying fields are not dynamicField... so "sku1" and "sku2" will be each copied to "1_s" and "2_s" respectively ... you could also mix & match that with a if you wanted sku1 and sku2 to have special types, but some ohther more common type for other sku* fields. : Regards, : Alex : : On Thu, Apr 5, 2018, 2:54 AM jatin roy, wrote: : : > Any update? : > : > From: jatin roy : > Sent: Tuesday, April 3, 2018 12:37 PM : > To: solr-user@lucene.apache.org : > Subject: Copy field on dynamic fields? : > : > Hi, : > : > Can we create copy field on dynamic fields? If yes then how it decide : > which field should be copied to which one? : > : > For example: if I have dynamic field: category_* and while indexing 4 : > fields are formed such as: : > category_1 : > category_2 : > category_3 : > category_4 : > and now I have to copy the contents of already existing dynamic field : > "category_*" to "new_category_*". : > : > So my question is how the algorithm decides that category_1 data has to be : > indexed in new_category_1 ? : > : > Regards : > Jatin Roy : > Software developer : > : > : -Hoss http://www.lucidworks.com/
Re: Copy field on dynamic fields?
Have you tried reading existing example schemas? They show various permutations of copy fields. Regards, Alex On Thu, Apr 5, 2018, 2:54 AM jatin roy, wrote: > Any update? > > From: jatin roy > Sent: Tuesday, April 3, 2018 12:37 PM > To: solr-user@lucene.apache.org > Subject: Copy field on dynamic fields? > > Hi, > > Can we create copy field on dynamic fields? If yes then how it decide > which field should be copied to which one? > > For example: if I have dynamic field: category_* and while indexing 4 > fields are formed such as: > category_1 > category_2 > category_3 > category_4 > and now I have to copy the contents of already existing dynamic field > "category_*" to "new_category_*". > > So my question is how the algorithm decides that category_1 data has to be > indexed in new_category_1 ? > > Regards > Jatin Roy > Software developer > >
Re: Copy field on dynamic fields?
Any update? From: jatin roy Sent: Tuesday, April 3, 2018 12:37 PM To: solr-user@lucene.apache.org Subject: Copy field on dynamic fields? Hi, Can we create copy field on dynamic fields? If yes then how it decide which field should be copied to which one? For example: if I have dynamic field: category_* and while indexing 4 fields are formed such as: category_1 category_2 category_3 category_4 and now I have to copy the contents of already existing dynamic field "category_*" to "new_category_*". So my question is how the algorithm decides that category_1 data has to be indexed in new_category_1 ? Regards Jatin Roy Software developer
Copy field on dynamic fields?
Hi, Can we create copy field on dynamic fields? If yes then how it decide which field should be copied to which one? For example: if I have dynamic field: category_* and while indexing 4 fields are formed such as: category_1 category_2 category_3 category_4 and now I have to copy the contents of already existing dynamic field "category_*" to "new_category_*". So my question is how the algorithm decides that category_1 data has to be indexed in new_category_1 ? Regards Jatin Roy Software developer
Re: Copy field and regex
On 12/8/2017 1:03 PM, Erick Erickson wrote: Second, grouping works fine in distributed mode with a couple of restrictions, see the reference guide. Collapse/Expand (an alternative to standard grouping) requires that all the members of a group be on the same shard. In 5.x, distributed grouping seems to require docValues, while in 4.x it didn't. I filed an issue about it. Yonik said that this is additional fallout from LUCENE-5666. I haven't tried it on 6.x or 7.x. https://issues.apache.org/jira/browse/SOLR-8088 If I send a request to one core, the grouping works, but if I make it distributed and the field doesn't have docValues, I get an exception. In order to accommodate the data-mining grouping queries I needed on my dev server (all production is still running 4.x versions), I used copyField to a string type with docValues and reindexed. I am not running SolrCloud for these indexes. Thanks, Shawn
Re: Copy field and regex
Grouping does _not_ require docValues, it's just that the with
docValues=false, uninverted structure is built on the heap at run
time. When docValues=true, the uninverted structure is written to disk
at index time and MMapped into the OS's memory space rather than the
Java heap.
Second, grouping works fine in distributed mode with a couple of
restrictions, see the reference guide. Collapse/Expand (an alternative
to standard grouping) requires that all the members of a group be on
the same shard.
Right, text fields aren't eligible for docValues, only "simple" types
(string included). If you want to use docValues, I'd recommend doing
the extraction on the client side. You can also put that in an update
component, but that's probably overkill.
Best,
Erick
On Fri, Dec 8, 2017 at 10:51 AM, Bradley Belyeu
wrote:
> Ah, thank you Erick & Shawn. That makes perfect sense. And yes when this goes
> to prod it will be distributed. Good point about docValues and needing a
> single shard, thanks!
> I’m new to result grouping, so I’m still prototyping that it will work for
> what I need.
>
> On 12/8/17, 12:00 PM, "Erick Erickson" wrote:
>
> I think you're getting confused by seeing the _stored_ data rather
> than the indexed data. When you return fields in documents, you get
> the stored data which is a verbatim copy of the input, no analysis
> done at all. To see what's in the index (and thus what would be
> grouped on) look at:
>
> adminUI>>analysis>>(your field) and put some sample values in and see
> what the regex transformer does. NOTE: unclick the "verbose" box for
> less clutter.
> or
> adminUI>>(select core)>>schema browser
> or
> termscomponent
>
> If you require the stored value to be different, you have several choices
> 1> change it on the client side before ingestion
> 2> use one of field mutating classes
>
> Most often, people don't bother storing the copyfield since the stored
> value is available in the original, the copyField destination is just
> used for things like you're interested in.
>
> Best,
> Erick
>
> On Fri, Dec 8, 2017 at 8:56 AM, Bradley Belyeu
> wrote:
> > I’m struggling a bit getting a copy field & regex tokenizer to work
> like I think it should…
> > I have an open source project I’m just starting out with here:
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fyouversion%2Fsolrcloud&data=02%7C01%7Cbradley.belyeu%40life.church%7C1c830048a2f84986e57d08d53e659b6d%7C8c9a6ca9b4314964afb4b8e1a2ba636f%7C1%7C0%7C636483528492765542&sdata=ZWo4gQwKOa0wGo5%2B822bro2sxnEg9F5b7cNil%2F0pj4k%3D&reserved=0
> > I have a uniqueKey field USFM defined as:
> > stored="true" />
> > And a USFM will always be in the pattern of 3 characters followed by a
> period followed by one or more digits followed by another period and finally
> one or more digits.
> > Optionally after the final digit there may be a hyphen and another
> digit.
> > IE: JHN.3.16 or MAT.6.33-34
> >
> > I’m wanting to do a result grouping by the first three characters,
> period, & digit(s). For example, docs with the unique keys JHN.3.16 &
> JHN.3.17 I would want grouped together.
> > So my thought was to define another field and then copy the USFM into
> it and use the regex tokenizer defined as so:
> >
> > positionIncrementGap="0">
> >
> > pattern="^(\w+\.\d+)\.\d+-*\d*$" group="1" />
> >
> >
> > stored="true" />
> >
> >
> > BUT, when I import my data the entire USFM is being stored inside the
> chapter field. And I get query results that look like:
> >{
> > "usfm":"MAT.10.1",
> > "chapter":"MAT.10.1",
> > "devo_keywords_en":"fear",
> > "_version_":1586184983451533312},
> > {
> > "usfm":"MAT.10.10",
> > "chapter":"MAT.10.10",
> > "devo_keywords_en":"fear",
> > "_version_":1586184983451533314},
> > {
> > "usfm":"MAT.10.11",
> > "chapter":"MAT.10.11",
> > "devo_keywords_en":"fear",
> > "_version_":1586184983451533316},
> > {
> > "usfm":"MAT.10.12",
> > "chapter":"MAT.10.12",
> > "devo_keywords_en":"fear",
> > "_version_":1586184983451533318}
> >
> > It’s probably something simple I’ve missed, but I’ve been banging my
> head for long enough I thought I’d ask for help.
> > Thanks in advance!
>
>
Re: Copy field and regex
Ah, thank you Erick & Shawn. That makes perfect sense. And yes when this goes
to prod it will be distributed. Good point about docValues and needing a single
shard, thanks!
I’m new to result grouping, so I’m still prototyping that it will work for what
I need.
On 12/8/17, 12:00 PM, "Erick Erickson" wrote:
I think you're getting confused by seeing the _stored_ data rather
than the indexed data. When you return fields in documents, you get
the stored data which is a verbatim copy of the input, no analysis
done at all. To see what's in the index (and thus what would be
grouped on) look at:
adminUI>>analysis>>(your field) and put some sample values in and see
what the regex transformer does. NOTE: unclick the "verbose" box for
less clutter.
or
adminUI>>(select core)>>schema browser
or
termscomponent
If you require the stored value to be different, you have several choices
1> change it on the client side before ingestion
2> use one of field mutating classes
Most often, people don't bother storing the copyfield since the stored
value is available in the original, the copyField destination is just
used for things like you're interested in.
Best,
Erick
On Fri, Dec 8, 2017 at 8:56 AM, Bradley Belyeu
wrote:
> I’m struggling a bit getting a copy field & regex tokenizer to work like
I think it should…
> I have an open source project I’m just starting out with here:
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fyouversion%2Fsolrcloud&data=02%7C01%7Cbradley.belyeu%40life.church%7C1c830048a2f84986e57d08d53e659b6d%7C8c9a6ca9b4314964afb4b8e1a2ba636f%7C1%7C0%7C636483528492765542&sdata=ZWo4gQwKOa0wGo5%2B822bro2sxnEg9F5b7cNil%2F0pj4k%3D&reserved=0
> I have a uniqueKey field USFM defined as:
>
> And a USFM will always be in the pattern of 3 characters followed by a
period followed by one or more digits followed by another period and finally
one or more digits.
> Optionally after the final digit there may be a hyphen and another digit.
> IE: JHN.3.16 or MAT.6.33-34
>
> I’m wanting to do a result grouping by the first three characters,
period, & digit(s). For example, docs with the unique keys JHN.3.16 & JHN.3.17
I would want grouped together.
> So my thought was to define another field and then copy the USFM into it
and use the regex tokenizer defined as so:
>
>
>
>
>
>
>
>
>
> BUT, when I import my data the entire USFM is being stored inside the
chapter field. And I get query results that look like:
>{
> "usfm":"MAT.10.1",
> "chapter":"MAT.10.1",
> "devo_keywords_en":"fear",
> "_version_":1586184983451533312},
> {
> "usfm":"MAT.10.10",
> "chapter":"MAT.10.10",
> "devo_keywords_en":"fear",
> "_version_":1586184983451533314},
> {
> "usfm":"MAT.10.11",
> "chapter":"MAT.10.11",
> "devo_keywords_en":"fear",
> "_version_":1586184983451533316},
> {
> "usfm":"MAT.10.12",
> "chapter":"MAT.10.12",
> "devo_keywords_en":"fear",
> "_version_":1586184983451533318}
>
> It’s probably something simple I’ve missed, but I’ve been banging my head
for long enough I thought I’d ask for help.
> Thanks in advance!
Re: Copy field and regex
I think you're getting confused by seeing the _stored_ data rather
than the indexed data. When you return fields in documents, you get
the stored data which is a verbatim copy of the input, no analysis
done at all. To see what's in the index (and thus what would be
grouped on) look at:
adminUI>>analysis>>(your field) and put some sample values in and see
what the regex transformer does. NOTE: unclick the "verbose" box for
less clutter.
or
adminUI>>(select core)>>schema browser
or
termscomponent
If you require the stored value to be different, you have several choices
1> change it on the client side before ingestion
2> use one of field mutating classes
Most often, people don't bother storing the copyfield since the stored
value is available in the original, the copyField destination is just
used for things like you're interested in.
Best,
Erick
On Fri, Dec 8, 2017 at 8:56 AM, Bradley Belyeu
wrote:
> I’m struggling a bit getting a copy field & regex tokenizer to work like I
> think it should…
> I have an open source project I’m just starting out with here:
> https://github.com/youversion/solrcloud
> I have a uniqueKey field USFM defined as:
> />
> And a USFM will always be in the pattern of 3 characters followed by a period
> followed by one or more digits followed by another period and finally one or
> more digits.
> Optionally after the final digit there may be a hyphen and another digit.
> IE: JHN.3.16 or MAT.6.33-34
>
> I’m wanting to do a result grouping by the first three characters, period, &
> digit(s). For example, docs with the unique keys JHN.3.16 & JHN.3.17 I would
> want grouped together.
> So my thought was to define another field and then copy the USFM into it and
> use the regex tokenizer defined as so:
>
>
>
> pattern="^(\w+\.\d+)\.\d+-*\d*$" group="1" />
>
>
> stored="true" />
>
>
> BUT, when I import my data the entire USFM is being stored inside the chapter
> field. And I get query results that look like:
>{
> "usfm":"MAT.10.1",
> "chapter":"MAT.10.1",
> "devo_keywords_en":"fear",
> "_version_":1586184983451533312},
> {
> "usfm":"MAT.10.10",
> "chapter":"MAT.10.10",
> "devo_keywords_en":"fear",
> "_version_":1586184983451533314},
> {
> "usfm":"MAT.10.11",
> "chapter":"MAT.10.11",
> "devo_keywords_en":"fear",
> "_version_":1586184983451533316},
> {
> "usfm":"MAT.10.12",
> "chapter":"MAT.10.12",
> "devo_keywords_en":"fear",
> "_version_":1586184983451533318}
>
> It’s probably something simple I’ve missed, but I’ve been banging my head for
> long enough I thought I’d ask for help.
> Thanks in advance!
Re: Copy field and regex
On 12/8/2017 9:56 AM, Bradley Belyeu wrote: > I’m wanting to do a result grouping by the first three characters, period, & > digit(s). For example, docs with the unique keys JHN.3.16 & JHN.3.17 I would > want grouped together. > So my thought was to define another field and then copy the USFM into it and > use the regex tokenizer defined as so: > > > > pattern="^(\w+\.\d+)\.\d+-*\d*$" group="1" /> > > > stored="true" /> > > > BUT, when I import my data the entire USFM is being stored inside the chapter > field. And I get query results that look like: Analysis only affects indexed terms. The field contents in query results is *ALWAYS* the original indexed text -- analysis *CANNOT* affect the fields returned for a document. The copyField feature does not copy the results of analysis, it always copies the original input. Since this is a "solr.TextField" type, you cannot define docValues on it, which means that the Result Grouping feature in Solr will use the indexed terms. Note that if your index is distributed, you probably won't be able to use the grouping feature -- that seems to require docValues. But if your index has a single shard, you should be OK. Thanks, Shawn
Copy field and regex
I’m struggling a bit getting a copy field & regex tokenizer to work like I
think it should…
I have an open source project I’m just starting out with here:
https://github.com/youversion/solrcloud
I have a uniqueKey field USFM defined as:
And a USFM will always be in the pattern of 3 characters followed by a period
followed by one or more digits followed by another period and finally one or
more digits.
Optionally after the final digit there may be a hyphen and another digit.
IE: JHN.3.16 or MAT.6.33-34
I’m wanting to do a result grouping by the first three characters, period, &
digit(s). For example, docs with the unique keys JHN.3.16 & JHN.3.17 I would
want grouped together.
So my thought was to define another field and then copy the USFM into it and
use the regex tokenizer defined as so:
BUT, when I import my data the entire USFM is being stored inside the chapter
field. And I get query results that look like:
{
"usfm":"MAT.10.1",
"chapter":"MAT.10.1",
"devo_keywords_en":"fear",
"_version_":1586184983451533312},
{
"usfm":"MAT.10.10",
"chapter":"MAT.10.10",
"devo_keywords_en":"fear",
"_version_":1586184983451533314},
{
"usfm":"MAT.10.11",
"chapter":"MAT.10.11",
"devo_keywords_en":"fear",
"_version_":1586184983451533316},
{
"usfm":"MAT.10.12",
"chapter":"MAT.10.12",
"devo_keywords_en":"fear",
"_version_":1586184983451533318}
It’s probably something simple I’ve missed, but I’ve been banging my head for
long enough I thought I’d ask for help.
Thanks in advance!
Re: Copy field from string to date
My apologies first for posting in the wrong forum as I thought they are internally using Solr and the the process is all same when it comes to filed mapping etc. And really appreciate your patience in assisting me. Thank you once again. -- View this message in context: http://lucene.472066.n3.nabble.com/Copy-field-from-string-to-date-tp4348173p4348303.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field from string to date
It would have been very helpful if you'd mentioned Fusion earlier. Fusion is a Lucidworks product, not something the Solr community can be expected to troubleshoot. I suggest you contact Lucidworks support for help with that. Best, Erick On Sun, Jul 30, 2017 at 11:56 AM, MKrishna wrote: > The actual setup is : > > Lucid works Fusion 3.1.0 (installed on a Linux machine) is the latest > upgrade we did for the search index from fusion 2.4.Also with 2.4 we had the > schema less mode. > > Basically we are indexing the outlook files stored in a folder via smb as > input to Lucid fusion. > > My application uses API to query the index and gets the result in JSON > format and applies additional condition before displaying to user. > > Please point me to any resources for mapping fields (assuming this must be > done before we index the data). > and what could be the issues with schema less? > > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Copy-field-from-string-to-date-tp4348173p4348298.html > Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field from string to date
The actual setup is : Lucid works Fusion 3.1.0 (installed on a Linux machine) is the latest upgrade we did for the search index from fusion 2.4.Also with 2.4 we had the schema less mode. Basically we are indexing the outlook files stored in a folder via smb as input to Lucid fusion. My application uses API to query the index and gets the result in JSON format and applies additional condition before displaying to user. Please point me to any resources for mapping fields (assuming this must be done before we index the data). and what could be the issues with schema less? -- View this message in context: http://lucene.472066.n3.nabble.com/Copy-field-from-string-to-date-tp4348173p4348298.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field from string to date
Version 3.1 of _what_? Certainly not Solr as schemaless is not in any 3x version of Solr. So it sounds like you changed some other part of your system. Can't help you there. And I do not recommend schemaless mode for production. Use it to find out what kinds of data you get, but then I'd advise you to map the fields produced from whatever software you're using to unpack your outlook into defined fields in Solr. Best, Erick On Sun, Jul 30, 2017 at 9:13 AM, MKrishna wrote: > We are using the schema less mode trying to parse outlook(.msg) files in an > smb. > and I get to see the properties of the .msg file(date_created fields ) > stored in dynamic fields date_Created_s and date_created_t. > > This change is seen only after upgrading the version to 3.1 as the older > versions(2.0*) still hold dates as date fields.Can you let me know if there > is anything to do with the parses or how do I check if the parser is right > ?? > > Thanks, > Krishna > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Copy-field-from-string-to-date-tp4348173p4348275.html > Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field from string to date
We are using the schema less mode trying to parse outlook(.msg) files in an smb. and I get to see the properties of the .msg file(date_created fields ) stored in dynamic fields date_Created_s and date_created_t. This change is seen only after upgrading the version to 3.1 as the older versions(2.0*) still hold dates as date fields.Can you let me know if there is anything to do with the parses or how do I check if the parser is right ?? Thanks, Krishna -- View this message in context: http://lucene.472066.n3.nabble.com/Copy-field-from-string-to-date-tp4348173p4348275.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field from string to date
It's right there in the reference guide: https://cwiki.apache.org/confluence/display/solr/Copying+Fields But that's not what I was talking about, you've got it backwards What you haven't really told us is why the data is being sent to date_created in the first place. How are you parsing the files? Why not just send the date to the right field in the first place? Best, Erick On Sat, Jul 29, 2017 at 10:47 PM, MKrishna wrote: > date_created field is being indexed as a string and text general making it > difficult to apply date range queries > > date_created_s and date_created_t are the extensions and trying to copy them > to *_tdate is throwing an error. > > "copyFields can take glob-type source specifications if that helps. " > > Can you give me an example for this as I want the text to be queried with > date queries? > > Thanks, > Krishna > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Copy-field-from-string-to-date-tp4348173p4348241.html > Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field from string to date
date_created field is being indexed as a string and text general making it difficult to apply date range queries date_created_s and date_created_t are the extensions and trying to copy them to *_tdate is throwing an error. "copyFields can take glob-type source specifications if that helps. " Can you give me an example for this as I want the text to be queried with date queries? Thanks, Krishna -- View this message in context: http://lucene.472066.n3.nabble.com/Copy-field-from-string-to-date-tp4348173p4348241.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field from string to date
bq: We want to use a copy field as a source for another copy field. As asked, this is not supported. You can copy the same source field to multiple copy fields however. copyFields can take glob-type source specifications if that helps. It would help if you gave concrete examples. You say "date created". How is that indexed? received_date ditto? If you have something like a dynamic field definition *_date of type, say, "tdate" you can copyField where the source is "*_date" for instance. Best, Erick On Sat, Jul 29, 2017 at 6:15 AM, MKrishna wrote: > We want to use a copy field as a source for another copy field. > > The problem is source field is from a text (dynamic field ) and > destination field should be date > tried changing the dynamic field datatype but it throws error. > > ideally my search index data source is outlook PST files as the index is > reading all date fields( properties of the files like date created, > received_date as text_general fields) making it difficult to query the > dates. > > All I want is to query the date fields that are stored as string > /text_general. > > Any help is really appreciated. > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Copy-field-from-string-to-date-tp4348173.html > Sent from the Solr - User mailing list archive at Nabble.com.
Copy field from string to date
We want to use a copy field as a source for another copy field. The problem is source field is from a text (dynamic field ) and destination field should be date tried changing the dynamic field datatype but it throws error. ideally my search index data source is outlook PST files as the index is reading all date fields( properties of the files like date created, received_date as text_general fields) making it difficult to query the dates. All I want is to query the date fields that are stored as string /text_general. Any help is really appreciated. -- View this message in context: http://lucene.472066.n3.nabble.com/Copy-field-from-string-to-date-tp4348173.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field a source of copy field
I get your point, the second KeepWordFilter is not keeping anything because the token it gets is : "hey you" and the word is supposed to keep is "hey". Which does clearly not work. The KeepWordFilter just consider each row a single token ( I may be wrong, i didn't check the code, I am just asssuming based on your observations). If you want, you can put a WordDelimiterFilter between the 2 KeepWordFilter. Configure the WordDelimiterFilter to split on space ( I need to double check, but it should be possible). OR You simply do as Erick suggested, and you just keep the genera in the genus field. But as Erick mentioned, you may have problems of entity recognition. - --- Alessandro Benedetti Search Consultant, R&D Software Engineer, Director Sease Ltd. - www.sease.io -- View this message in context: http://lucene.472066.n3.nabble.com/Copy-field-a-source-of-copy-field-tp4346425p4347731.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field a source of copy field
Je, I also think that!. We have some serious gaps on what you explain to me. First, you point me that there's no real need to use ShingleFilter, I tried with all Tokenizer and the result is the same, the species are not caught. On the simplest scenario I've got this: PUT YOUR FAVORITE TOKENIZER HERE And testing on Analysis tab, wouldn't catch any tag with blank space, like "acacia acicularis". Am I missing something? Then, by using ShingleFilter, tags with blank space are caught correctly. But you said you're having no trouble applying multiple successive keepword filters. So, I just use 2 KWF files as I depict: On species file there's only one line, that is "hey you" on genus file, there's also one line, which is "hey" Catching nothing at all for the second KWF <http://lucene.472066.n3.nabble.com/file/n4347541/1.png> Well, I have to say I'm so confused with this behaviour, have I forgot something? -- View this message in context: http://lucene.472066.n3.nabble.com/Copy-field-a-source-of-copy-field-tp4346425p4347541.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field a source of copy field
Yep, we're not communication ;) Use the original source field for the genus, as: The difficulty here is that there might be false hits if the genera names happen to match words in the input that are not part of a genus/species pair. On Thu, Jul 20, 2017 at 9:55 AM, tstusr wrote: > Well, correct me if I'm wrong. > > Your suggestion is to use species field as a source of genus field. We try > with this > > > > > Where species work as described and genus just use a KWF, like this: > > positionIncrementGap="0"> > > >ignoreCase="true"/> > > > > > > > > But now, the problem now is different. > > When we try the behavior in analysis section in solr provided UI it works as > expected. > > Nevertheless, when we use it at indexing time (When we post pdf files, to > extractor) the field doesn't even appear. We think it's because the info > becomes from another copyField. > > Did I misunderstand your suggestion? > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Copy-field-a-source-of-copy-field-tp4346425p4347013.html > Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field a source of copy field
Well, correct me if I'm wrong. Your suggestion is to use species field as a source of genus field. We try with this Where species work as described and genus just use a KWF, like this: But now, the problem now is different. When we try the behavior in analysis section in solr provided UI it works as expected. Nevertheless, when we use it at indexing time (When we post pdf files, to extractor) the field doesn't even appear. We think it's because the info becomes from another copyField. Did I misunderstand your suggestion? -- View this message in context: http://lucene.472066.n3.nabble.com/Copy-field-a-source-of-copy-field-tp4346425p4347013.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field a source of copy field
OK, you'll need two fields pretty much for certain. The trick is getting _only_ genus names in the genus field. The simplest thing to do would be a straight copyField with a single keep word filter that contains a list of all the genera. That presupposes that the genera are disjoint sets from all other words. You search on your species field and facet on the genus field. But assuming your genera are not disjoint from all other words, h. Do you have a way of unambiguously identifying genus/species pairs in the text you're processing? If you do we can work with that, but without that you're talking entity recognition of some sort. BTW, there's no real need to shingle the species field, just search for "genus species" as a phrase. Unless those two appear next to each other in order you won't get a hit. Best, Erick On Wed, Jul 19, 2017 at 11:07 AM, tstusr wrote: > Well, our documents consist on pdf files (between 20 to 200 pages). > > So, we catch words of all the file, for that, we use the extract handler, > that's why we have this fields: > > > > > We catch species in all the pdf content (On attr_content field) > > Species captured are used for ranking purposes. So, we have to have the > whole name, that's why we use shingles. As an example, we catch from the > pdf: > > abelmoschus achanioides > abies colimensis > abies concolor > > Because that information is important, we provide a facet of those species, > grouped by genus (just the first word of the species). So, in the facet we > have to have: > > abelmoschus (1) > abies (2) > > Nevertheless, we need a sort of subquery, because first, we need the > complete species and then of those results facet by genus. For example: > > the abies something else (This phrase shouldn't have to be captured) > the abies concolor something else (This phrase should've to be captured) -> > Finish with just "abies concolor" and for consequence then captured by genus > > I realized that all genus are contained on species. > > So, there is a way to make a facet with just the first word of a field, like > I've got for the field: > > abelmoschus achanioides > abies colimensis > abies concolor > > Just use the first word of those? > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Copy-field-a-source-of-copy-field-tp4346425p4346846.html > Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field a source of copy field
Well, our documents consist on pdf files (between 20 to 200 pages). So, we catch words of all the file, for that, we use the extract handler, that's why we have this fields: We catch species in all the pdf content (On attr_content field) Species captured are used for ranking purposes. So, we have to have the whole name, that's why we use shingles. As an example, we catch from the pdf: abelmoschus achanioides abies colimensis abies concolor Because that information is important, we provide a facet of those species, grouped by genus (just the first word of the species). So, in the facet we have to have: abelmoschus (1) abies (2) Nevertheless, we need a sort of subquery, because first, we need the complete species and then of those results facet by genus. For example: the abies something else (This phrase shouldn't have to be captured) the abies concolor something else (This phrase should've to be captured) -> Finish with just "abies concolor" and for consequence then captured by genus I realized that all genus are contained on species. So, there is a way to make a facet with just the first word of a field, like I've got for the field: abelmoschus achanioides abies colimensis abies concolor Just use the first word of those? -- View this message in context: http://lucene.472066.n3.nabble.com/Copy-field-a-source-of-copy-field-tp4346425p4346846.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field a source of copy field
OK, I take it back. Keepwords handle multiple words just fine. So I have to rewind. I'm having no trouble at all applying multiple, successive keepwords filters, even when there are multiple words on a single line in the keepwords file. Your use of shingles in here is probably going to confuse things, so I'd probably recommend taking that out until you work out what's happening with multiple keepwords filters, then add it back in. The images you pasted almost look like you're showing the contents of elevate.xml, but I suspect that's bogus. But I think this is an XY problem, you're asking about how to chain copyFields and we got off into talking about chaining keepwords and the like. You state: "So, the requirements here, are to be able to find all species in species files (step one) and then make a facet with species in file genus, step two." Then you say: "And the second one (genus), which contains genus that has to be for facet purposes, like this" How are those reconciled? Do you want facets on the genus+species? Or just on the genus? Or both? So let's just start over. What's also missing is why you think you need keepwords in the first place. Is this a free-text field you're trying to extract genus/species from? Or do you have the genus/species extracted already? Give us two docs, a sample search and what you want as outcome. Because if you just want to facet on genus then do a copyField simply to a "genus" field that strips out everything but the genus (however you implement that, tricky given sub-species perhaps). Ditto if you want to facet on species. Just a species_facet field that you put whatever you want into. Or just use KeywordTokenizer for species if you're guaranteed that you want the whole field. You can then use copyField to copy as you wish. Best, Erick On Tue, Jul 18, 2017 at 2:23 PM, tstusr wrote: > Well, for me it's kind of strange because it's working only with words that > have blank spaces. It seems that maybe I'm not explaining well. > > My field is defined as follows: > >positionIncrementGap="0"> > > >mapping="mapping/mapping-ISOLatin1Accent.txt"/> >pattern="[0-9]+|(\-)(\s*)" replacement=""/> > >outputUnigrams="true"/> >ignoreCase="true"/> >ignoreCase="true"/> > > > > > > > > We have 2 KWF files, "species" and then "genus". It seems that is just > working with genus. > > Since I'm not able to use copy fields, what choices I have? > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Copy-field-a-source-of-copy-field-tp4346425p4346665.html > Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field a source of copy field
Well, for me it's kind of strange because it's working only with words that have blank spaces. It seems that maybe I'm not explaining well. My field is defined as follows: We have 2 KWF files, "species" and then "genus". It seems that is just working with genus. Since I'm not able to use copy fields, what choices I have? -- View this message in context: http://lucene.472066.n3.nabble.com/Copy-field-a-source-of-copy-field-tp4346425p4346665.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field a source of copy field
Multiple keyword files work just fine for me. one issue you're having is that multi-word keepwords aren't going to do what you expect. The analysis chains work on _tokens_, and only see one at a time. Plus (apparently) the input is broken up on whitespace (the docs aren't entirely clear on this, but can be inferred by "one per line"). Even if there were multi-word keepwords, it wouldn't work as you apparently expect. The problem is that the analysis chain first breaks the input into tokens. So even if a "single" keepword were "a b", and your input was "a b", by the time it gets to the keepword filter the context would be lost. So the filter would see just "a" and say "nope it doesn't match 'a b', throw it out". Ditto with "b". Since keepwords are apparently split on whitespace though, in the example above both would be kept. The keepword list is "a" and "b" so in the above example both match and are kept. Best, Erkck On Tue, Jul 18, 2017 at 9:49 AM, tstusr wrote: > Well, I have no idea why that images display as did. > > The correct order is: > > Field chain analyzer. > <http://lucene.472066.n3.nabble.com/file/n4346602/1.png> > > KWF-genus file > <http://lucene.472066.n3.nabble.com/file/n4346602/3.png> > > Test output. > <http://lucene.472066.n3.nabble.com/file/n4346602/2.png> > > Sorry for the mistake > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Copy-field-a-source-of-copy-field-tp4346425p4346602.html > Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field a source of copy field
Well, I have no idea why that images display as did. The correct order is: Field chain analyzer. <http://lucene.472066.n3.nabble.com/file/n4346602/1.png> KWF-genus file <http://lucene.472066.n3.nabble.com/file/n4346602/3.png> Test output. <http://lucene.472066.n3.nabble.com/file/n4346602/2.png> Sorry for the mistake -- View this message in context: http://lucene.472066.n3.nabble.com/Copy-field-a-source-of-copy-field-tp4346425p4346602.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field a source of copy field
It seems that is just taking the last file of keep words. <http://lucene.472066.n3.nabble.com/file/n4346601/Screen_Shot_2017-07-18_at_11.png> Now for control purposes, I have in genus file: <http://lucene.472066.n3.nabble.com/file/n4346601/Screen_Shot_2017-07-18_at_11.png> And just is taking the composed field, abutilon aurantiacum. By testing with abutilon aurantiacum abutilon bakerianum <http://lucene.472066.n3.nabble.com/file/n4346601/Screen_Shot_2017-07-18_at_11.png> It's is not possible to put 2 tokenizers in a field, am I right? Because I just think there is a missing split in between the 2 KWFs. -- View this message in context: http://lucene.472066.n3.nabble.com/Copy-field-a-source-of-copy-field-tp4346425p4346601.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field a source of copy field
The code is very simple, it looks at a quick glance like it just reads the words in then the "accept" method just returns true or false based on whether the text file contains the token. Are you sure you reloaded your core/collection and pushed the changed schema to the right place? The admin/analysis page is very helpful here, your indexing side should have two keep word filters and you should be able to see each transformation (uncheck the "verbose" checkbox for more readability. Best, Erick On Tue, Jul 18, 2017 at 8:49 AM, tstusr wrote: > Ok, I know shingling will join with "_". > > But that is the behaviour we want, imagine we have this fields (contained in > species file): > > abarema idiopoda > abutilon bakerianum > > Those become in: > abarema > idiopoda > abutilon > bakerianum > abarema_idiopoda > abutilon_bakerianum > > But now in my genus file maybe is only the word abarema, so, we end up with > a field with only that word. > > So, the requirements here, are to be able to find all species in species > files (step one) and then make a facet with species in file genus, step two. > > It seems reasonable to just chain the fields, I just forgot solr didn't > change the field, as Shawn points (thanks for it). > > So what we came here is to make 2 fields the first with species. > > positionIncrementGap="0"> > > >mapping="mapping/mapping-ISOLatin1Accent.txt"/> >pattern="[0-9]+|(\-)(\s*)" replacement=""/> > >outputUnigrams="true"/> >ignoreCase="true"/> > > > >outputUnigrams="false"/> > > > > > And the second one (genus), which contains genus that has to be for facet > purposes, like this: > > positionIncrementGap="0"> > > >mapping="mapping/mapping-ISOLatin1Accent.txt"/> >pattern="[0-9]+|(\-)(\s*)" replacement=""/> > > outputUnigrams="true"/> >ignoreCase="true"/> >ignoreCase="true"/> > > > > > > > > Nevertheless, there is no second processing for keep word filter as (I) > expect. Am I missing something? > > > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Copy-field-a-source-of-copy-field-tp4346425p4346593.html > Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field a source of copy field
Ok, I know shingling will join with "_". But that is the behaviour we want, imagine we have this fields (contained in species file): abarema idiopoda abutilon bakerianum Those become in: abarema idiopoda abutilon bakerianum abarema_idiopoda abutilon_bakerianum But now in my genus file maybe is only the word abarema, so, we end up with a field with only that word. So, the requirements here, are to be able to find all species in species files (step one) and then make a facet with species in file genus, step two. It seems reasonable to just chain the fields, I just forgot solr didn't change the field, as Shawn points (thanks for it). So what we came here is to make 2 fields the first with species. And the second one (genus), which contains genus that has to be for facet purposes, like this: Nevertheless, there is no second processing for keep word filter as (I) expect. Am I missing something? -- View this message in context: http://lucene.472066.n3.nabble.com/Copy-field-a-source-of-copy-field-tp4346425p4346593.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Copy field a source of copy field
On 7/17/2017 4:26 PM, tstusr wrote: > We want to use a copy field as a source for another copy field or some kind > of post processing of a field. > As an example imagine we have on species > > abies durangensis > abies flinckii > > so, after post processing, we expect to have only > abies > > which is a word in genus files Let's say that you have this in your schema, and you index "Test Words" (note the capital letters) in field a: Let's say that the index analysis on field a has the whitespace tokenizer, a lowercase filter, and a stopword filter with "test" in the list. This means that the search terms for field a on that document will only have "words" included. You might be expecting field b to only receive "words" when it gets copied from field a ... but this is NOT what happens. Field b receives the original text sent to field a, which is "Test Words", including both words and the uppercase letters. I think that transitive copies *do* work, so that you can copy field a to b, then field b to c, though I am not 100 percent sure about that. If that does work, the end field in the chain is still going to receive "Test Words" like you sent to field a. Chaining analysis through copyField does not work. Thanks, Shawn
Re: Copy field a source of copy field
In a word, "no". Copyfields are not chained together. I'm not at all sure what you're trying to accomplish with those filter chains anyway, By shingling _then_ doing the stopwords, you'll have some input like abies durangensis become abies abies_durangensis durangensis Then put that through your keepwords filter which presumably only has species in it so it would throw out abies and abies_durangensis unless those are in your keepwords file Seems a waste. That aside, you can construct one long analysis chain that combined the genus and species chains and just copy from attr_content* into both. You wouldn't get the different tokenization, but presumably you don't particularly need it on the second part of the chain. Best, Erick On Mon, Jul 17, 2017 at 3:26 PM, tstusr wrote: > Hi > > We want to use a copy field as a source for another copy field or some kind > of post processing of a field. > > The problem is here. We have a field from a text that is captured by a > field, like this: > > > > which has (at the end of the processing) just the words in a field. > > termVectors="true" termPositions="true" termOffsets="true"/> > > positionIncrementGap="0"> > > >mapping="mapping/mapping-ISOLatin1Accent.txt"/> >pattern="[0-9]+|(\-)(\s*)" replacement=""/> > >outputUnigrams="true"/> >ignoreCase="true"/> > > > >outputUnigrams="false"/> > > > > > So, what we want to do now is to implement a faceting according to some post > processing of this field by using this as a source for another field. > > > > positionIncrementGap="0"> > > > >ignoreCase="true"/> > > > > > > > > > As soon as I understand. We don't have a value on genus because the chain is > ended. Nevertheless, we are also not available to make two processings to > first, capture the words on species and then make a new capture for the > genus. > > As an example imagine we have on species > > abies durangensis > abies flinckii > > so, after post processing, we expect to have only > abies > > which is a word in genus files > > I was as clear as possible with the problem, but maybe there are some black > holes in the explanation. > > Hope you can help me. > > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Copy-field-a-source-of-copy-field-tp4346425.html > Sent from the Solr - User mailing list archive at Nabble.com.
Copy field a source of copy field
Hi We want to use a copy field as a source for another copy field or some kind of post processing of a field. The problem is here. We have a field from a text that is captured by a field, like this: which has (at the end of the processing) just the words in a field. So, what we want to do now is to implement a faceting according to some post processing of this field by using this as a source for another field. As soon as I understand. We don't have a value on genus because the chain is ended. Nevertheless, we are also not available to make two processings to first, capture the words on species and then make a new capture for the genus. As an example imagine we have on species abies durangensis abies flinckii so, after post processing, we expect to have only abies which is a word in genus files I was as clear as possible with the problem, but maybe there are some black holes in the explanation. Hope you can help me. -- View this message in context: http://lucene.472066.n3.nabble.com/Copy-field-a-source-of-copy-field-tp4346425.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Atomic Update w/ Date Copy Field
Stefan Matheis-3 wrote
> To me, it sounds more like you shouldn’t have to care about such gory
> details as a user - at all.
>
> would you mind opening a issue on JIRA Todd? Including all the details you
> already provided in as well as a link to this thread, would be best.
>
> Depending on what you actually did to find this all out, you probably do
> even have a test case at hand which demonstrates the behaviour? if not,
> that’s obviously not a problem :)
Agreed on the gory details. Yes, it definitely seems like the format should
be consistent between full and partial updates. I'll go ahead and open an
issue on JIRA.
Alexandre Rafalovitch wrote
> I noticed (and abused) the issue Todd described in my Solr puzzle at:
> http://blog.outerthoughts.com/2016/04/solr-5-puzzle-magic-date-answer/
>
> The second format ("EEE...") looks rather strange. I would suspect
> that the conversion Date->String code is using the active locale and
> that is the format default for that locale. So, the bug might be that
> the locale needs to be more specific to preserve the consistence.
Thank you for the Solr puzzle reference. The EEE format is most certainly
the java.util.Date.toString() method being called when re-creating the
field.
--
View this message in context:
http://lucene.472066.n3.nabble.com/Atomic-Update-w-Date-Copy-Field-tp4293779p4295049.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Atomic Update w/ Date Copy Field
I noticed (and abused) the issue Todd described in my Solr puzzle at:
http://blog.outerthoughts.com/2016/04/solr-5-puzzle-magic-date-answer/
The second format ("EEE...") looks rather strange. I would suspect
that the conversion Date->String code is using the active locale and
that is the format default for that locale. So, the bug might be that
the locale needs to be more specific to preserve the consistence.
Regards,
Alex.
Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/
On 30 August 2016 at 23:25, Stefan Matheis wrote:
> To me, it sounds more like you shouldn’t have to care about such gory details
> as a user - at all.
>
> would you mind opening a issue on JIRA Todd? Including all the details you
> already provided in as well as a link to this thread, would be best.
>
> Depending on what you actually did to find this all out, you probably do even
> have a test case at hand which demonstrates the behaviour? if not, that’s
> obviously not a problem :)
>
> -Stefan
>
>
> On August 30, 2016 at 3:51:42 PM, Todd Long (lon...@gmail.com) wrote:
>> It looks like the issue has to do with the Date object. When the document is
>> fully updated (with the date specified) the field is created with a String
>> object so everything is indexed as it appears. When the document is
>> partially updated (with the date omitted) the field is re-created using the
>> previously stored Date object which takes the "toString" representation
>> (i.e. EEE MMM dd HH:mm:ss zzz ).
>>
>> I ended up creating a DateTextField which extends TextField and simply
>> overrides the "FieldType.createField(SchemaField, Object, float)" method. I
>> then check for a Date instance and format as necessary.
>>
>> Any ideas on a better approach or does it sound like this is the way to go?
>> I wasn't sure if this could be accomplished in a filter or some other way.
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/Atomic-Update-w-Date-Copy-Field-tp4293779p4293968.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>
Re: Atomic Update w/ Date Copy Field
To me, it sounds more like you shouldn’t have to care about such gory details as a user - at all. would you mind opening a issue on JIRA Todd? Including all the details you already provided in as well as a link to this thread, would be best. Depending on what you actually did to find this all out, you probably do even have a test case at hand which demonstrates the behaviour? if not, that’s obviously not a problem :) -Stefan On August 30, 2016 at 3:51:42 PM, Todd Long (lon...@gmail.com) wrote: > It looks like the issue has to do with the Date object. When the document is > fully updated (with the date specified) the field is created with a String > object so everything is indexed as it appears. When the document is > partially updated (with the date omitted) the field is re-created using the > previously stored Date object which takes the "toString" representation > (i.e. EEE MMM dd HH:mm:ss zzz ). > > I ended up creating a DateTextField which extends TextField and simply > overrides the "FieldType.createField(SchemaField, Object, float)" method. I > then check for a Date instance and format as necessary. > > Any ideas on a better approach or does it sound like this is the way to go? > I wasn't sure if this could be accomplished in a filter or some other way. > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Atomic-Update-w-Date-Copy-Field-tp4293779p4293968.html > > Sent from the Solr - User mailing list archive at Nabble.com. >
Re: Atomic Update w/ Date Copy Field
It looks like the issue has to do with the Date object. When the document is fully updated (with the date specified) the field is created with a String object so everything is indexed as it appears. When the document is partially updated (with the date omitted) the field is re-created using the previously stored Date object which takes the "toString" representation (i.e. EEE MMM dd HH:mm:ss zzz ). I ended up creating a DateTextField which extends TextField and simply overrides the "FieldType.createField(SchemaField, Object, float)" method. I then check for a Date instance and format as necessary. Any ideas on a better approach or does it sound like this is the way to go? I wasn't sure if this could be accomplished in a filter or some other way. -- View this message in context: http://lucene.472066.n3.nabble.com/Atomic-Update-w-Date-Copy-Field-tp4293779p4293968.html Sent from the Solr - User mailing list archive at Nabble.com.
Atomic Update w/ Date Copy Field
We recently started using atomic updates in our application and have since
noticed that date fields copied to a text field have varying results between
full and partial updates. When the document is fully updated the copied text
date appears as expected (i.e. -MM-dd'T'HH:mm:ss.SSSZ); however, when
the document is partially updated (while omitting the date field) the
original stored date value is copied to a different format (i.e. EEE MMM d
HH:mm:ss z ). I've included an example below of what we are seeing with
the indexed value of our "createdDate_facet_t" field. Is there a way that we
can force the copy field to always use "-MM-dd'T'HH:mm:ss.SSSZ" as the
resulting text format without having to always include the field in the
update?
schema
/update (full)
-
{
"id": "12345",
"createdBy_t": "someone",
"createdDate_dt": "2015-07-14T12:58:17.535Z"
}
createdDate_facet_t = "2015-07-14t12:58:17.535z"
/update (partial)
{
"id": "12345",
"createdBy_t": { "set": "another" }
}
createdDate_facet_t = "tue jul 14 12:58:17 utc 2015"
--
View this message in context:
http://lucene.472066.n3.nabble.com/Atomic-Update-w-Date-Copy-Field-tp4293779.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: copy field source not working in solr schema.xml
Although what you pasted isn't the complete schema I guess you miss a wrote: > Error : > org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: > Could not load conf for core demo7: copyField dest :'i_member_id' is not an > explicit field and doesn't match a dynamicField.. Schema file is > /opt/solr/example/solr/demo7/conf/schema.xml > > My schema.xml : > > required="true" multiValued="false" /> > stored="true" required="false" /> > > > > > Please help me. > Thanks in advance. > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/copy-field-source-not-working-in-solr-schema-xml-tp4273355.html > Sent from the Solr - User mailing list archive at Nabble.com. >
copy field source not working in solr schema.xml
Error : org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Could not load conf for core demo7: copyField dest :'i_member_id' is not an explicit field and doesn't match a dynamicField.. Schema file is /opt/solr/example/solr/demo7/conf/schema.xml My schema.xml : Please help me. Thanks in advance. -- View this message in context: http://lucene.472066.n3.nabble.com/copy-field-source-not-working-in-solr-schema-xml-tp4273355.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr auggestion with copy-field
There are always some steps to take care of in these situations : 1) have you checked that your destination copied field is fine ? Is it containing what you expect ? have you investigated the indexed terms ? 2) Have you built your suggester ? It doesn't build on startup or onCommit ( reading your config) , so you should build it to see the suggestions Hard to believe it depends on your copyfield , from the Suggester perspective , it doesn't know if the field configured is coming from a copy field or not. Cheers 2015-09-21 11:34 GMT+01:00 sara hajili : > hi all > i wanna to get suggestion from multi field in solr. > i add this to solrConfig > > > mySuggester > FuzzyLookupFactory > DocumentDictionaryFactory > suggestStr > like_count > string > false > > > > > true > 10 > > > suggest > > > > and add this to schema: > > and > termVectors="true"/> >termVectors="true"/> >termVectors="true"/> > multiValued="true" termVectors="true" /> > > but i didn't get any result from this suggest query: > > http://localhost:8983/solr/post/suggest?suggest=true&suggest.dictionary=mySuggester&wt=json&suggest.q=solr > > but when i used one field (not copy field ) i got answer. > how i solve my problem with copy field? > -- -- Benedetti Alessandro Visiting card - http://about.me/alessandro_benedetti Blog - http://alexbenedetti.blogspot.co.uk "Tyger, tyger burning bright In the forests of the night, What immortal hand or eye Could frame thy fearful symmetry?" William Blake - Songs of Experience -1794 England
solr auggestion with copy-field
hi all i wanna to get suggestion from multi field in solr. i add this to solrConfig mySuggester FuzzyLookupFactory DocumentDictionaryFactory suggestStr like_count string false true 10 suggest and add this to schema: and but i didn't get any result from this suggest query: http://localhost:8983/solr/post/suggest?suggest=true&suggest.dictionary=mySuggester&wt=json&suggest.q=solr but when i used one field (not copy field ) i got answer. how i solve my problem with copy field?
Re: A defect in Schema API with Add a New Copy Field Rule?
Thanks Steve and Yonik. This now makes sense. Updating the doc will be of a big help. With regards to deleting a copy-field, what I found is that if I have N instances of the same copy-field, I have to issue N deletes to remove them all. This behavior matches with the add and need to be kept. Steve On Wed, May 6, 2015 at 8:10 PM, Steve Rowe wrote: > Hi Steve, > > It’s by design that you can copyField the same source/dest multiple times > - according to Yonik (not sure where this was discussed), this capability > has been used in the past to effectively boost terms in the source field. > > The API isn’t symmetric here though: I’m guessing deleting a mutiply > specified copy field rule will delete all of them, but this isn’t tested, > so I’m not sure. > > There is no replace-copy-field command because copy field rules don’t have > dependencies (i.e., nothing else in the schema refers to copy field rules), > unlike fields, dynamic fields and field types, so > delete-copy-field/add-copy-field works as one would expect. > > For fields, dynamic fields and field types, a delete followed by an add is > not the same as a replace, since (dynamic) fields could have dependent > copyFields, and field types could have dependent (dynamic) fields. > delete-* commands are designed to fail if there are any existing > dependencies, while the replace-* commands will maintain the dependencies > if they exist. > > Steve > > > On May 6, 2015, at 6:44 PM, Steven White wrote: > > > > Hi Everyone, > > > > I am using the Schema API to add a new copy field per: > > > https://cwiki.apache.org/confluence/display/solr/Schema+API#SchemaAPI-AddaNewCopyFieldRule > > > > Unlike the other "Add" APIs, this one will not fail if you add an > existing > > copy field object. In fact, after when I call the API over and over, the > > item will appear over and over in schema.xml file like so: > > > > > > > > > > > > > > Is this the expected behaviour or a bug? As a side question, is there > any > > harm in having multiple "copyField" like I ended up with? > > > > A final question, why there is no Replace a Copy Field? Is this by > design > > for some limitation or was the API just never implemented? > > > > Thanks > > > > Steve > >
Re: A defect in Schema API with Add a New Copy Field Rule?
> On May 6, 2015, at 8:25 PM, Yonik Seeley wrote: > > On Wed, May 6, 2015 at 8:10 PM, Steve Rowe wrote: >> It’s by design that you can copyField the same source/dest multiple times - >> according to Yonik (not sure where this was discussed), this capability has >> been used in the past to effectively boost terms in the source field. > > Yep, used to be relatively common. > Perhaps the API could be cleaner though if we supported that by > passing an optional "numTimes" or "numCopies"? Seems like a sane > delete / overwrite options would thus be easier? +1
Re: A defect in Schema API with Add a New Copy Field Rule?
On Wed, May 6, 2015 at 8:10 PM, Steve Rowe wrote: > It’s by design that you can copyField the same source/dest multiple times - > according to Yonik (not sure where this was discussed), this capability has > been used in the past to effectively boost terms in the source field. Yep, used to be relatively common. Perhaps the API could be cleaner though if we supported that by passing an optional "numTimes" or "numCopies"? Seems like a sane delete / overwrite options would thus be easier? -Yonik
Re: A defect in Schema API with Add a New Copy Field Rule?
Hi Steve, It’s by design that you can copyField the same source/dest multiple times - according to Yonik (not sure where this was discussed), this capability has been used in the past to effectively boost terms in the source field. The API isn’t symmetric here though: I’m guessing deleting a mutiply specified copy field rule will delete all of them, but this isn’t tested, so I’m not sure. There is no replace-copy-field command because copy field rules don’t have dependencies (i.e., nothing else in the schema refers to copy field rules), unlike fields, dynamic fields and field types, so delete-copy-field/add-copy-field works as one would expect. For fields, dynamic fields and field types, a delete followed by an add is not the same as a replace, since (dynamic) fields could have dependent copyFields, and field types could have dependent (dynamic) fields. delete-* commands are designed to fail if there are any existing dependencies, while the replace-* commands will maintain the dependencies if they exist. Steve > On May 6, 2015, at 6:44 PM, Steven White wrote: > > Hi Everyone, > > I am using the Schema API to add a new copy field per: > https://cwiki.apache.org/confluence/display/solr/Schema+API#SchemaAPI-AddaNewCopyFieldRule > > Unlike the other "Add" APIs, this one will not fail if you add an existing > copy field object. In fact, after when I call the API over and over, the > item will appear over and over in schema.xml file like so: > > > > > > > Is this the expected behaviour or a bug? As a side question, is there any > harm in having multiple "copyField" like I ended up with? > > A final question, why there is no Replace a Copy Field? Is this by design > for some limitation or was the API just never implemented? > > Thanks > > Steve
A defect in Schema API with Add a New Copy Field Rule?
Hi Everyone, I am using the Schema API to add a new copy field per: https://cwiki.apache.org/confluence/display/solr/Schema+API#SchemaAPI-AddaNewCopyFieldRule Unlike the other "Add" APIs, this one will not fail if you add an existing copy field object. In fact, after when I call the API over and over, the item will appear over and over in schema.xml file like so: Is this the expected behaviour or a bug? As a side question, is there any harm in having multiple "copyField" like I ended up with? A final question, why there is no Replace a Copy Field? Is this by design for some limitation or was the API just never implemented? Thanks Steve
Re: Order of Copy Field and Analyzer
Yes, the "copied" value is always the original one (the stored), regardless any analysis, which is field-scoped On 23 Apr 2015 19:13, "Kaushik" wrote: > Hello, > > > What is the order in which these occur? > > >- Copy field >- Analyzer > > The other way of asking the above question I guess is, if I copy an _txt > field to _t field, does the analyzer of _t get the orignial text sent to > _txt field or the analyzed tokens from it? > > > Thanks, > > Kaushik >
Order of Copy Field and Analyzer
Hello, What is the order in which these occur? - Copy field - Analyzer The other way of asking the above question I guess is, if I copy an _txt field to _t field, does the analyzer of _t get the orignial text sent to _txt field or the analyzed tokens from it? Thanks, Kaushik
Re: sort by a copy field error
On 4/15/2015 2:02 AM, Pedro Figueiredo wrote: > My solr installation is in cloud mode... so the basic solr stop and start > does not update the configuration right? > > I started solr using: > solr -c -Dbootstrap_confdir=C:\solr-5.0.0\server\solr\patientsCollection\conf > -Dcollection.configName=myconf > > and the error was solved. I really dislike the bootstrap options. They are designed to be run exactly once -- when you are first converting a non-cloud config to cloud. If they are used beyond that, they will likely cause confusion. When you're running SolrCloud, editing the schema on the disk does nothing. You must upload the changed config to zookeeper, where SolrCloud actually looks for config data. That's what the startup options that you added will do, is upload that specific config to zookeeper with that specific name, and it will do it every time you start Solr with those options. The zkcli script, specifically its "upconfig" command, should be used to do that. Then you can use the Collections API to reload the collection, and follow that up with a reindex. https://cwiki.apache.org/confluence/display/solr/Command+Line+Utilities https://cwiki.apache.org/confluence/display/solr/Collections+API Thanks, Shawn
RE: sort by a copy field error
Ok... my bad...
My solr installation is in cloud mode... so the basic solr stop and start does
not update the configuration right?
I started solr using:
solr -c -Dbootstrap_confdir=C:\solr-5.0.0\server\solr\patientsCollection\conf
-Dcollection.configName=myconf
and the error was solved.
Please, advice if this is the correct command to update the solr configuration
in cloud mode.
Thanks,
Pedro Figueiredo
Senior Engineer
pjlfigueir...@criticalsoftware.com
M. 934058150
Rua Engº Frederico Ulrich, nº 2650 4470-605 Moreira da Maia, Portugal
T. +351 229 446 927 | F. +351 229 446 929
www.criticalsoftware.com
PORTUGAL | UK | GERMANY | USA | BRAZIL | MOZAMBIQUE | ANGOLA
A CMMI® LEVEL 5 RATED COMPANY CMMI® is registered in the USPTO by CMU"
-Original Message-
From: Andrea Gazzarini [mailto:a.gazzar...@gmail.com]
Sent: 15 April 2015 08:41
To: solr-user@lucene.apache.org
Subject: Re: sort by a copy field error
Really strange to me: the cause should be what Shawn already pointed out,
because that error is raised when:
SchemaField sf = req.getSchema().getFieldOrNull(field);
is null:
if (null == sf) {
...
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "sort param
field can't be found: " + field); }
So it seems Solr doesn't find the "name_sort" field in the schema, as you
changed that (the schema) without reloading / restarting.
Andrea
On 04/15/2015 09:30 AM, Pedro Figueiredo wrote:
> Hello,
>
> http://localhost:8983/solr/patientsCollection/select?q=*%3A*&sort=name
> _sort+asc&wt=json&indent=true&_=1429082874881
>
> I am using the solr console admin and in the "query" option I just define the
> field sort with "name_sort asc".
>
> Pedro Figueiredo
> Senior Engineer
>
> pjlfigueir...@criticalsoftware.com
> M. 934058150
>
>
> Rua Engº Frederico Ulrich, nº 2650 4470-605 Moreira da Maia, Portugal
> T. +351 229 446 927 | F. +351 229 446 929 www.criticalsoftware.com
>
> PORTUGAL | UK | GERMANY | USA | BRAZIL | MOZAMBIQUE | ANGOLA A CMMI®
> LEVEL 5 RATED COMPANY CMMI® is registered in the USPTO by CMU"
>
>
>
> -Original Message-
> From: Andrea Gazzarini [mailto:a.gazzar...@gmail.com]
> Sent: 14 April 2015 19:47
> To: solr-user@lucene.apache.org
> Subject: Re: sort by a copy field error
>
> Hi Pedro
> Please post the request that produces that error
>
> Andrea
> On 14 Apr 2015 19:33, "Pedro Figueiredo"
>
> wrote:
>
>> Hello,
>>
>>
>>
>> I have a pretty basic question: how can I sort by a copyfield?
>>
>>
>>
>> My schema conf is:
>>
>>
>>
>> > stored="true" omitNorms="true" termVectors="true"/>
>>
>>
>>
>>
>>
>>
>>
>> And when I try to sort by "name_sort" the following error is raised:
>>
>> "error": {
>>
>> "msg": "sort param field can't be found: name_sort",
>>
>> "code": 400
>>
>>}
>>
>>
>>
>> Thanks in advanced,
>>
>>
>>
>> Pedro Figueiredo
>>
>>
>>
>>
Re: sort by a copy field error
Really strange to me: the cause should be what Shawn already pointed
out, because that error is raised when:
SchemaField sf = req.getSchema().getFieldOrNull(field);
is null:
if (null == sf) {
...
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "sort
param field can't be found: " + field);
}
So it seems Solr doesn't find the "name_sort" field in the schema, as
you changed that (the schema) without reloading / restarting.
Andrea
On 04/15/2015 09:30 AM, Pedro Figueiredo wrote:
Hello,
http://localhost:8983/solr/patientsCollection/select?q=*%3A*&sort=name_sort+asc&wt=json&indent=true&_=1429082874881
I am using the solr console admin and in the "query" option I just define the field sort
with "name_sort asc".
Pedro Figueiredo
Senior Engineer
pjlfigueir...@criticalsoftware.com
M. 934058150
Rua Engº Frederico Ulrich, nº 2650 4470-605 Moreira da Maia, Portugal
T. +351 229 446 927 | F. +351 229 446 929
www.criticalsoftware.com
PORTUGAL | UK | GERMANY | USA | BRAZIL | MOZAMBIQUE | ANGOLA
A CMMI® LEVEL 5 RATED COMPANY CMMI® is registered in the USPTO by CMU"
-Original Message-
From: Andrea Gazzarini [mailto:a.gazzar...@gmail.com]
Sent: 14 April 2015 19:47
To: solr-user@lucene.apache.org
Subject: Re: sort by a copy field error
Hi Pedro
Please post the request that produces that error
Andrea
On 14 Apr 2015 19:33, "Pedro Figueiredo"
wrote:
Hello,
I have a pretty basic question: how can I sort by a copyfield?
My schema conf is:
And when I try to sort by "name_sort" the following error is raised:
"error": {
"msg": "sort param field can't be found: name_sort",
"code": 400
}
Thanks in advanced,
Pedro Figueiredo
RE: sort by a copy field error
Hello,
http://localhost:8983/solr/patientsCollection/select?q=*%3A*&sort=name_sort+asc&wt=json&indent=true&_=1429082874881
I am using the solr console admin and in the "query" option I just define the
field sort with "name_sort asc".
Pedro Figueiredo
Senior Engineer
pjlfigueir...@criticalsoftware.com
M. 934058150
Rua Engº Frederico Ulrich, nº 2650 4470-605 Moreira da Maia, Portugal
T. +351 229 446 927 | F. +351 229 446 929
www.criticalsoftware.com
PORTUGAL | UK | GERMANY | USA | BRAZIL | MOZAMBIQUE | ANGOLA
A CMMI® LEVEL 5 RATED COMPANY CMMI® is registered in the USPTO by CMU"
-Original Message-
From: Andrea Gazzarini [mailto:a.gazzar...@gmail.com]
Sent: 14 April 2015 19:47
To: solr-user@lucene.apache.org
Subject: Re: sort by a copy field error
Hi Pedro
Please post the request that produces that error
Andrea
On 14 Apr 2015 19:33, "Pedro Figueiredo"
wrote:
> Hello,
>
>
>
> I have a pretty basic question: how can I sort by a copyfield?
>
>
>
> My schema conf is:
>
>
>
> stored="true" omitNorms="true" termVectors="true"/>
>
>
>
>
>
>
>
> And when I try to sort by "name_sort" the following error is raised:
>
> "error": {
>
> "msg": "sort param field can't be found: name_sort",
>
> "code": 400
>
> }
>
>
>
> Thanks in advanced,
>
>
>
> Pedro Figueiredo
>
>
>
>
RE: sort by a copy field error
Hello,
Yes I restart solr and re-index after the change.
The request is:
http://localhost:8983/solr/patientsCollection/select?q=*%3A*&sort=name_sort+asc&wt=json&indent=true&_=1429082874881
I am using the solr console admin and in the "query" option I just define the
field sort with "name_sort asc".
thanks!
Pedro Figueiredo
Senior Engineer
pjlfigueir...@criticalsoftware.com
M. 934058150
Rua Engº Frederico Ulrich, nº 2650 4470-605 Moreira da Maia, Portugal
T. +351 229 446 927 | F. +351 229 446 929
www.criticalsoftware.com
PORTUGAL | UK | GERMANY | USA | BRAZIL | MOZAMBIQUE | ANGOLA
A CMMI® LEVEL 5 RATED COMPANY CMMI® is registered in the USPTO by CMU"
-Original Message-
From: Shawn Heisey [mailto:apa...@elyograg.org]
Sent: 14 April 2015 19:44
To: solr-user@lucene.apache.org
Subject: Re: sort by a copy field error
On 4/14/2015 11:32 AM, Pedro Figueiredo wrote:
> And when I try to sort by "name_sort" the following error is raised:
>
> "error": {
>
> "msg": "sort param field can't be found: name_sort",
>
> "code": 400
>
> }
What was the exact sort parameter you sent to Solr?
Did you reload the core or restart Solr and then reindex after you changed your
schema? A reindex will be required.
http://wiki.apache.org/solr/HowToReindex
Thanks,
Shawn
Re: sort by a copy field error
Hi Pedro
Please post the request that produces that error
Andrea
On 14 Apr 2015 19:33, "Pedro Figueiredo"
wrote:
> Hello,
>
>
>
> I have a pretty basic question: how can I sort by a copyfield?
>
>
>
> My schema conf is:
>
>
>
> stored="true" omitNorms="true" termVectors="true"/>
>
>
>
>
>
>
>
> And when I try to sort by "name_sort" the following error is raised:
>
> "error": {
>
> "msg": "sort param field can't be found: name_sort",
>
> "code": 400
>
> }
>
>
>
> Thanks in advanced,
>
>
>
> Pedro Figueiredo
>
>
>
>
Re: sort by a copy field error
On 4/14/2015 11:32 AM, Pedro Figueiredo wrote:
> And when I try to sort by "name_sort" the following error is raised:
>
> "error": {
>
> "msg": "sort param field can't be found: name_sort",
>
> "code": 400
>
> }
What was the exact sort parameter you sent to Solr?
Did you reload the core or restart Solr and then reindex after you
changed your schema? A reindex will be required.
http://wiki.apache.org/solr/HowToReindex
Thanks,
Shawn
sort by a copy field error
Hello,
I have a pretty basic question: how can I sort by a copyfield?
My schema conf is:
And when I try to sort by "name_sort" the following error is raised:
"error": {
"msg": "sort param field can't be found: name_sort",
"code": 400
}
Thanks in advanced,
Pedro Figueiredo
Re: copy field from boolean to int
I already use this field elsewhere, so I don't want to change it's type. I did implement a UpdateRequestProcessor to copy from a bool to an int. This works, but even better would be to fix Solr so that I can use DocValues with boolean. So, I am going to try to get that working as well. On Tue, Mar 17, 2015 at 10:25 PM, William Bell wrote: > Can you reindex? Just use 1,0. > > On Tue, Mar 17, 2015 at 6:08 PM, Chris Hostetter > > wrote: > > > > > Can you open a jira to add docValues support for BoolField? ... i can't > > think of any good reason not to directly support that in Solr for > > BoolField ... seems like just an oversight that slipped through the > > cracks. > > > > > > For now, your best bet is probably to use an UpdateProcessor ... maybe 2 > > instances of RegexReplaceProcessorFactory to match "true" and "false" and > > replace them with "0" and "1" ? > > > > > > : Date: Tue, 17 Mar 2015 17:57:03 -0700 > > : From: Kevin Osborn > > : Reply-To: solr-user@lucene.apache.org > > : To: solr-user@lucene.apache.org > > : Subject: copy field from boolean to int > > : > > : I was hoping to use DocValues, but one of my fields is a boolean, which > > is > > : not currently supported by DocValues. I can use a copyField to convert > my > > : boolean to a string. Is there is anyway to use a copyField to convert > > from > > : a boolean to a tint? > > > > > > -Hoss > > http://www.lucidworks.com/ > > > > > > -- > Bill Bell > billnb...@gmail.com > cell 720-256-8076 >
Re: copy field from boolean to int
Can you reindex? Just use 1,0. On Tue, Mar 17, 2015 at 6:08 PM, Chris Hostetter wrote: > > Can you open a jira to add docValues support for BoolField? ... i can't > think of any good reason not to directly support that in Solr for > BoolField ... seems like just an oversight that slipped through the > cracks. > > > For now, your best bet is probably to use an UpdateProcessor ... maybe 2 > instances of RegexReplaceProcessorFactory to match "true" and "false" and > replace them with "0" and "1" ? > > > : Date: Tue, 17 Mar 2015 17:57:03 -0700 > : From: Kevin Osborn > : Reply-To: solr-user@lucene.apache.org > : To: solr-user@lucene.apache.org > : Subject: copy field from boolean to int > : > : I was hoping to use DocValues, but one of my fields is a boolean, which > is > : not currently supported by DocValues. I can use a copyField to convert my > : boolean to a string. Is there is anyway to use a copyField to convert > from > : a boolean to a tint? > > > -Hoss > http://www.lucidworks.com/ > -- Bill Bell billnb...@gmail.com cell 720-256-8076
Re: copy field from boolean to int
Can you open a jira to add docValues support for BoolField? ... i can't think of any good reason not to directly support that in Solr for BoolField ... seems like just an oversight that slipped through the cracks. For now, your best bet is probably to use an UpdateProcessor ... maybe 2 instances of RegexReplaceProcessorFactory to match "true" and "false" and replace them with "0" and "1" ? : Date: Tue, 17 Mar 2015 17:57:03 -0700 : From: Kevin Osborn : Reply-To: solr-user@lucene.apache.org : To: solr-user@lucene.apache.org : Subject: copy field from boolean to int : : I was hoping to use DocValues, but one of my fields is a boolean, which is : not currently supported by DocValues. I can use a copyField to convert my : boolean to a string. Is there is anyway to use a copyField to convert from : a boolean to a tint? -Hoss http://www.lucidworks.com/
copy field from boolean to int
I was hoping to use DocValues, but one of my fields is a boolean, which is not currently supported by DocValues. I can use a copyField to convert my boolean to a string. Is there is anyway to use a copyField to convert from a boolean to a tint?
Re: Can I define the copy field like title_*
thank you very much Mr. Sumit -- View this message in context: http://lucene.472066.n3.nabble.com/Can-I-define-the-copy-field-like-title-tp468p471.html Sent from the Solr - User mailing list archive at Nabble.com.
Can I define the copy field like title_*
I just wonder can I define copy field like this instead of millions thank you Rachun -- View this message in context: http://lucene.472066.n3.nabble.com/Can-I-define-the-copy-field-like-title-tp468.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Can I define the copy field like title_*
Yes it's legitimate to copy like - Sumit Arora -- View this message in context: http://lucene.472066.n3.nabble.com/Can-I-define-the-copy-field-like-title-tp468p470.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Copy field append values ?
Appended. -- Jack Krupansky -Original Message- From: vishgupt Sent: Wednesday, October 16, 2013 6:25 AM To: solr-user@lucene.apache.org Subject: Solr Copy field append values ? Hi , Schema like this external_id is multivalued field. I want to know will values of upc will be appended to exiting values of external_id or override it ? For example if I send a document having values upc:131 external_id:423 for indexing in sorl with above mentioned schema.what will be value of external_id field 131 or 131,423. Thanks Vishal -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Copy-field-append-values-tp4095862.html Sent from the Solr - User mailing list archive at Nabble.com.
Solr Copy field append values ?
Hi , Schema like this external_id is multivalued field. I want to know will values of upc will be appended to exiting values of external_id or override it ? For example if I send a document having values upc:131 external_id:423 for indexing in sorl with above mentioned schema.what will be value of external_id field 131 or 131,423. Thanks Vishal -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Copy-field-append-values-tp4095862.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: SOLR Copy field if no value on destination
Oh yeah. Hi have seen that Processor on the book and i was not able to remember. Thanks a lot. And thanks a lot for your solution. It works :) On Aug 8, 2013, at 1:52 AM, "Jack Krupansky" wrote: > Here's the actual update processor I used (and tested): > > > > main_s > final_s > > > backup_s > final_s > > > final_s > > > > > > -- Jack Krupansky > > -Original Message- From: Jack Krupansky > Sent: Wednesday, August 07, 2013 8:20 PM > To: solr-user@lucene.apache.org > Subject: Re: SOLR Copy field if no value on destination > > Sorry, I am unable to untangle the logic you are expressing, but I can can > assure you that JavaScript and the StatelessScriptUpdate processor has full > support for implementing spaghetti code logic as tangled as desired! > > Simpler forms of logic can be implemented directly using non-script update > processor sequences, but once you start adding conditionals, there is a 50% > chance that you will need a script. > > There is a Default Value update processor, but it takes a literal value. > > Hmmm... maybe I’ll come up with a “default-value” script that takes a field > name for the default value. IOW, it would copy a specified field to the > destination IFF the destination had no value. > > Ahhh... wait... maybe... you could do this with the First Value Update > processor: > > 1. Copy guid to FinalLink. (Clone Update processor). > 2. Copy link to FinalLink. (Clone Update processor). > 3. First Value Update processor. > > So, step 3 would leave link if guid was not there, or keep guid if it is > there and discard link. > > Yes, that should do it. > > This is worth an example in the book! Thanks for the inspiration! > > -- Jack Krupansky > > From: Luís Portela Afonso > Sent: Wednesday, August 07, 2013 7:22 PM > To: solr-user@lucene.apache.org > Subject: SOLR Copy field if no value on destination > > Hi, > > Is possible to copy a value of a field to another if the destination doesn't > have value? > An example: > a.. Indexing an rss > b.. The feed has the fields link and guid, but sometimes guid cannot be > present in the feed > c.. I have a field that i will copy values with the name finalLink > > Now i want to copy guid to finalLink, but if guid has not value i want to > copy link. > > My question is, is that possible just with the schema, Processors, > solrconfig.xml, and the data-config? > > Thanks a lot smime.p7s Description: S/MIME cryptographic signature
Re: SOLR Copy field if no value on destination
Here's the actual update processor I used (and tested): main_s final_s backup_s final_s final_s -- Jack Krupansky -Original Message- From: Jack Krupansky Sent: Wednesday, August 07, 2013 8:20 PM To: solr-user@lucene.apache.org Subject: Re: SOLR Copy field if no value on destination Sorry, I am unable to untangle the logic you are expressing, but I can can assure you that JavaScript and the StatelessScriptUpdate processor has full support for implementing spaghetti code logic as tangled as desired! Simpler forms of logic can be implemented directly using non-script update processor sequences, but once you start adding conditionals, there is a 50% chance that you will need a script. There is a Default Value update processor, but it takes a literal value. Hmmm... maybe I’ll come up with a “default-value” script that takes a field name for the default value. IOW, it would copy a specified field to the destination IFF the destination had no value. Ahhh... wait... maybe... you could do this with the First Value Update processor: 1. Copy guid to FinalLink. (Clone Update processor). 2. Copy link to FinalLink. (Clone Update processor). 3. First Value Update processor. So, step 3 would leave link if guid was not there, or keep guid if it is there and discard link. Yes, that should do it. This is worth an example in the book! Thanks for the inspiration! -- Jack Krupansky From: Luís Portela Afonso Sent: Wednesday, August 07, 2013 7:22 PM To: solr-user@lucene.apache.org Subject: SOLR Copy field if no value on destination Hi, Is possible to copy a value of a field to another if the destination doesn't have value? An example: a.. Indexing an rss b.. The feed has the fields link and guid, but sometimes guid cannot be present in the feed c.. I have a field that i will copy values with the name finalLink Now i want to copy guid to finalLink, but if guid has not value i want to copy link. My question is, is that possible just with the schema, Processors, solrconfig.xml, and the data-config? Thanks a lot
Re: SOLR Copy field if no value on destination
Sorry, I am unable to untangle the logic you are expressing, but I can can assure you that JavaScript and the StatelessScriptUpdate processor has full support for implementing spaghetti code logic as tangled as desired! Simpler forms of logic can be implemented directly using non-script update processor sequences, but once you start adding conditionals, there is a 50% chance that you will need a script. There is a Default Value update processor, but it takes a literal value. Hmmm... maybe I’ll come up with a “default-value” script that takes a field name for the default value. IOW, it would copy a specified field to the destination IFF the destination had no value. Ahhh... wait... maybe... you could do this with the First Value Update processor: 1. Copy guid to FinalLink. (Clone Update processor). 2. Copy link to FinalLink. (Clone Update processor). 3. First Value Update processor. So, step 3 would leave link if guid was not there, or keep guid if it is there and discard link. Yes, that should do it. This is worth an example in the book! Thanks for the inspiration! -- Jack Krupansky From: Luís Portela Afonso Sent: Wednesday, August 07, 2013 7:22 PM To: solr-user@lucene.apache.org Subject: SOLR Copy field if no value on destination Hi, Is possible to copy a value of a field to another if the destination doesn't have value? An example: a.. Indexing an rss b.. The feed has the fields link and guid, but sometimes guid cannot be present in the feed c.. I have a field that i will copy values with the name finalLink Now i want to copy guid to finalLink, but if guid has not value i want to copy link. My question is, is that possible just with the schema, Processors, solrconfig.xml, and the data-config? Thanks a lot
Re: SOLR Copy field if no value on destination
Yes, it is possible to copy from a field to another field that has no value. In fact, that is the only kind of copy you should be doing unless the field is multivalued. IOW, copy field is not “replace field”. -- Jack Krupansky From: Luís Portela Afonso Sent: Wednesday, August 07, 2013 7:22 PM To: solr-user@lucene.apache.org Subject: SOLR Copy field if no value on destination Hi, Is possible to copy a value of a field to another if the destination doesn't have value? An example: a.. Indexing an rss b.. The feed has the fields link and guid, but sometimes guid cannot be present in the feed c.. I have a field that i will copy values with the name finalLink Now i want to copy guid to finalLink, but if guid has not value i want to copy link. My question is, is that possible just with the schema, Processors, solrconfig.xml, and the data-config? Thanks a lot
SOLR Copy field if no value on destination
Hi, Is possible to copy a value of a field to another if the destination doesn't have value? An example: Indexing an rss The feed has the fields link and guid, but sometimes guid cannot be present in the feed I have a field that i will copy values with the name finalLink Now i want to copy guid to finalLink, but if guid has not value i want to copy link. My question is, is that possible just with the schema, Processors, solrconfig.xml, and the data-config? Thanks a lot smime.p7s Description: S/MIME cryptographic signature
Re: copy Field / postprocess Fields after analyze / dynamic analyzer config
Not that I know of, sorry. Best Erick On Fri, Feb 8, 2013 at 10:11 AM, Kai Gülzau wrote: > I there a way to postprocess a field after analyze? > > Saying postprocess I think of renaming, moving or appending fields. > > > Some more information: > > My schema.xml contains several language suffixed fields (nouns_de, ...). > Each of these is analyzed in a language dependent way: > > positionIncrementGap="100"> > >descriptorPath="/uima/AggregateSentenceDEAE.xml" > tokenType="org.apache.uima.TokenAnnotation" featurePath="posTag"/> > types="/uima/whitelist_de.txt" /> > > > > > > > When I do a facted search I have to include every field_lang combination > since I do not know the language at query time: > > > http://localhost:8983/solr/master/select?q=*:*&rows=0&facet=true&facet.field=nouns_de&facet.field=nouns_en&facet.field=nouns_fr&facet.field=nouns_nl... > > So I have to merge all terms in my own business logic :-( > > > Any idea / pointer to rename fields after analyze? > > This post says it's not possible with the current API: > http://lucene.472066.n3.nabble.com/copyField-after-analyzer-td3900337.html > > > Another approach would be to allow analyzer configuration depending on > another field value (language). > > > regards, > > Kai Gülzau > >
copy Field / postprocess Fields after analyze / dynamic analyzer config
I there a way to postprocess a field after analyze? Saying postprocess I think of renaming, moving or appending fields. Some more information: My schema.xml contains several language suffixed fields (nouns_de, ...). Each of these is analyzed in a language dependent way: When I do a facted search I have to include every field_lang combination since I do not know the language at query time: http://localhost:8983/solr/master/select?q=*:*&rows=0&facet=true&facet.field=nouns_de&facet.field=nouns_en&facet.field=nouns_fr&facet.field=nouns_nl ... So I have to merge all terms in my own business logic :-( Any idea / pointer to rename fields after analyze? This post says it's not possible with the current API: http://lucene.472066.n3.nabble.com/copyField-after-analyzer-td3900337.html Another approach would be to allow analyzer configuration depending on another field value (language). regards, Kai Gülzau
Re: Copy Field Question
Hello, I think you don't have that much tuning possiblities using only the schema.xml file. You will have to write some custom Java code (subclasses of UpdateRequestProcessor and UpdateRequestProcessorFactory), build a Java jar containing your custom code, put that jar in one of the path declared your solrconfig.xml ( ) -- or add a new one, and finally tune the update processors chain configuration (still in solrconfig.xml) so your custom update processor is used. See http://wiki.apache.org/solr/UpdateRequestProcessor which uses exactly your use case as an example. I hope this will help you :) -- Tanguy 2012/10/15 Virendra Goswami > Can we limit copyfield source condition? > for example if we want to make lookup in source="product_name" and > dest="some_dest" > so our syntax would become > > How about copying only those product_names having status=0 AND attribute1=1 > AND attribute2=0. > assume status,attribute1,attribute2 and product_name being two different > attribute of a same table. > can we write something like > source="attribute:1" AND source="attribute2:0" dest="some_dest" > maxchar=200> > > Thanks in advance >
Copy Field Question
Can we limit copyfield source condition? for example if we want to make lookup in source="product_name" and dest="some_dest" so our syntax would become How about copying only those product_names having status=0 AND attribute1=1 AND attribute2=0. assume status,attribute1,attribute2 and product_name being two different attribute of a same table. can we write something like Thanks in advance
Re: How to boost score of particular field in a copy field?
Have you tried simply searching across all 30 fields using (e)dismax? You can set this up in a request handler in solrconfig.xml and forget about it. You could also de-boost the copied-to field by using a boost < 1 (but > 0, e.g. 0.5). Best Erick On Fri, Aug 17, 2012 at 10:13 AM, bbarani wrote: > Thanks a lot for your reply!! > > This is the problem I face now when using dismax. > > I have almost 30 fields (Some are dynamic fields from database) copied in to > copy field that is being used for default search. Now when I search using > dismax I need to include the boost for all the fields in the search query > else it returns only the result corresponding to the field in qf. > > http://localhost:8983/solr/db/select?q=Stanley,%20robert&defType=dismax&qf=name > > I get just one result for above query > > http://localhost:8983/solr/db/select?q=Stanley,%20robert&defType=dismax&qf=name > lastupdatedBy > > Now the search happens across 2 fields. > > I want to boost a particular field but at the same time I want the results > that matches keyword from other fileds too. > > Thanks, > BB > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/How-to-boost-score-of-particular-field-in-a-copy-field-tp4001864p4001873.html > Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to boost score of particular field in a copy field?
Thanks a lot for your reply!! This is the problem I face now when using dismax. I have almost 30 fields (Some are dynamic fields from database) copied in to copy field that is being used for default search. Now when I search using dismax I need to include the boost for all the fields in the search query else it returns only the result corresponding to the field in qf. http://localhost:8983/solr/db/select?q=Stanley,%20robert&defType=dismax&qf=name I get just one result for above query http://localhost:8983/solr/db/select?q=Stanley,%20robert&defType=dismax&qf=name lastupdatedBy Now the search happens across 2 fields. I want to boost a particular field but at the same time I want the results that matches keyword from other fileds too. Thanks, BB -- View this message in context: http://lucene.472066.n3.nabble.com/How-to-boost-score-of-particular-field-in-a-copy-field-tp4001864p4001873.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to boost score of particular field in a copy field?
What you should consider is to use the edismax (or dismax) query parser and then you can boost the individual fields. That's a major part of its reason for existence - to overcome the limitation of copyField that you mention. -- Jack Krupansky -Original Message- From: bbarani Sent: Friday, August 17, 2012 11:47 AM To: solr-user@lucene.apache.org Subject: How to boost score of particular field in a copy field? Hi, I have 10 fields copied in to a single copy field for default search. I am trying to boost one particular field in that copy field, but I am not sure how boosting in copy field work. Can someone let me know how boosting works in copy field? Thanks, BB -- View this message in context: http://lucene.472066.n3.nabble.com/How-to-boost-score-of-particular-field-in-a-copy-field-tp4001864.html Sent from the Solr - User mailing list archive at Nabble.com.
How to boost score of particular field in a copy field?
Hi, I have 10 fields copied in to a single copy field for default search. I am trying to boost one particular field in that copy field, but I am not sure how boosting in copy field work. Can someone let me know how boosting works in copy field? Thanks, BB -- View this message in context: http://lucene.472066.n3.nabble.com/How-to-boost-score-of-particular-field-in-a-copy-field-tp4001864.html Sent from the Solr - User mailing list archive at Nabble.com.
