Re: Solr dynamic "on the fly fields"
Some aggregations are supported by combining stats with pivot facets? See: https://lucidworks.com/2015/01/29/you-got-stats-in-my-facets/ Don't quite think that works for your use case though. the other thing that _might_ help is all the Streaming Expression/Streaming Aggregation work. Best, Erick On Wed, Jul 5, 2017 at 6:23 AM, Pablo Anzorenawrote: > Thanks Erick for the answer. Function Queries are great, but for my use > case what I really do is making aggregations (using Json Facet for example) > with this functions. > > I have tried using Function Queries with Json Facet but it does not support > it. > > Any other idea you can imagine? > > > > > > 2017-07-03 21:57 GMT-03:00 Erick Erickson : > >> I don't know how one would do this. But I would ask what the use-case >> is. Creating such fields at index time just seems like it would be >> inviting abuse by creating a zillion fields as you have no control >> over what gets created. I'm assuming your tenants don't talk to each >> other >> >> Have you thought about using function queries to pull this data out as >> needed at _query_ time? See: >> https://cwiki.apache.org/confluence/display/solr/Function+Queries >> >> Best, >> Erick >> >> On Mon, Jul 3, 2017 at 12:06 PM, Pablo Anzorena >> wrote: >> > Thanks Erick, >> > >> > For my use case it's not possible any of those solutions. I have a >> > multitenancy scheme in the most basic level, that is I have a single >> > collection with fields (clientId, field1, field2, ..., field50) attending >> > many clients. >> > >> > Clients can create custom fields based on arithmetic operations of any >> > other field. >> > >> > So, is it possible to update let's say field49 with the follow operation: >> > log(field39) + field25 on clientId=43? >> > >> > Do field39 and field25 need to be stored to accomplish this? Is there any >> > other way to avoid storing them? >> > >> > Thanks! >> > >> > >> > 2017-07-03 15:00 GMT-03:00 Erick Erickson : >> > >> >> There are two ways: >> >> 1> define a dynamic field pattern, i.e. >> >> >> >> >> >> >> >> Now just add any field in the doc you want. If it ends in "_sum" and >> >> no other explicit field matches you have a new field. >> >> >> >> 2> Use the managed schema to add these on the fly. I don't recommend >> >> this from what I know of your use case, this is primarily intended for >> >> front-ends to be able to modify the schema and/or "field guessing". >> >> >> >> I do caution you though that either way don't go over-the-top. If >> >> you're thinking of thousands of different fields that can lead to >> >> performance issues. >> >> >> >> You can either put stuff in the field on your indexing client or >> >> create a custom update component, perhaps the simplest would be a >> >> "StatelessScriptUpdateProcessorFactory: >> >> >> >> see: https://cwiki.apache.org/confluence/display/solr/ >> >> Update+Request+Processors#UpdateRequestProcessors- >> >> UpdateRequestProcessorFactories >> >> >> >> Best, >> >> Erick >> >> >> >> On Mon, Jul 3, 2017 at 10:52 AM, Pablo Anzorena < >> anzorena.f...@gmail.com> >> >> wrote: >> >> > Hey, >> >> > >> >> > I was wondering if there is some way to add fields "on the fly" based >> on >> >> > arithmetic operations on other fields. For example add a new field >> >> > "custom_field" = log(field1) + field2 -5. >> >> > >> >> > Thanks. >> >> >>
Re: Solr dynamic "on the fly fields"
Thanks Erick for the answer. Function Queries are great, but for my use case what I really do is making aggregations (using Json Facet for example) with this functions. I have tried using Function Queries with Json Facet but it does not support it. Any other idea you can imagine? 2017-07-03 21:57 GMT-03:00 Erick Erickson: > I don't know how one would do this. But I would ask what the use-case > is. Creating such fields at index time just seems like it would be > inviting abuse by creating a zillion fields as you have no control > over what gets created. I'm assuming your tenants don't talk to each > other > > Have you thought about using function queries to pull this data out as > needed at _query_ time? See: > https://cwiki.apache.org/confluence/display/solr/Function+Queries > > Best, > Erick > > On Mon, Jul 3, 2017 at 12:06 PM, Pablo Anzorena > wrote: > > Thanks Erick, > > > > For my use case it's not possible any of those solutions. I have a > > multitenancy scheme in the most basic level, that is I have a single > > collection with fields (clientId, field1, field2, ..., field50) attending > > many clients. > > > > Clients can create custom fields based on arithmetic operations of any > > other field. > > > > So, is it possible to update let's say field49 with the follow operation: > > log(field39) + field25 on clientId=43? > > > > Do field39 and field25 need to be stored to accomplish this? Is there any > > other way to avoid storing them? > > > > Thanks! > > > > > > 2017-07-03 15:00 GMT-03:00 Erick Erickson : > > > >> There are two ways: > >> 1> define a dynamic field pattern, i.e. > >> > >> > >> > >> Now just add any field in the doc you want. If it ends in "_sum" and > >> no other explicit field matches you have a new field. > >> > >> 2> Use the managed schema to add these on the fly. I don't recommend > >> this from what I know of your use case, this is primarily intended for > >> front-ends to be able to modify the schema and/or "field guessing". > >> > >> I do caution you though that either way don't go over-the-top. If > >> you're thinking of thousands of different fields that can lead to > >> performance issues. > >> > >> You can either put stuff in the field on your indexing client or > >> create a custom update component, perhaps the simplest would be a > >> "StatelessScriptUpdateProcessorFactory: > >> > >> see: https://cwiki.apache.org/confluence/display/solr/ > >> Update+Request+Processors#UpdateRequestProcessors- > >> UpdateRequestProcessorFactories > >> > >> Best, > >> Erick > >> > >> On Mon, Jul 3, 2017 at 10:52 AM, Pablo Anzorena < > anzorena.f...@gmail.com> > >> wrote: > >> > Hey, > >> > > >> > I was wondering if there is some way to add fields "on the fly" based > on > >> > arithmetic operations on other fields. For example add a new field > >> > "custom_field" = log(field1) + field2 -5. > >> > > >> > Thanks. > >> >
Re: Solr dynamic "on the fly fields"
I don't know how one would do this. But I would ask what the use-case is. Creating such fields at index time just seems like it would be inviting abuse by creating a zillion fields as you have no control over what gets created. I'm assuming your tenants don't talk to each other Have you thought about using function queries to pull this data out as needed at _query_ time? See: https://cwiki.apache.org/confluence/display/solr/Function+Queries Best, Erick On Mon, Jul 3, 2017 at 12:06 PM, Pablo Anzorenawrote: > Thanks Erick, > > For my use case it's not possible any of those solutions. I have a > multitenancy scheme in the most basic level, that is I have a single > collection with fields (clientId, field1, field2, ..., field50) attending > many clients. > > Clients can create custom fields based on arithmetic operations of any > other field. > > So, is it possible to update let's say field49 with the follow operation: > log(field39) + field25 on clientId=43? > > Do field39 and field25 need to be stored to accomplish this? Is there any > other way to avoid storing them? > > Thanks! > > > 2017-07-03 15:00 GMT-03:00 Erick Erickson : > >> There are two ways: >> 1> define a dynamic field pattern, i.e. >> >> >> >> Now just add any field in the doc you want. If it ends in "_sum" and >> no other explicit field matches you have a new field. >> >> 2> Use the managed schema to add these on the fly. I don't recommend >> this from what I know of your use case, this is primarily intended for >> front-ends to be able to modify the schema and/or "field guessing". >> >> I do caution you though that either way don't go over-the-top. If >> you're thinking of thousands of different fields that can lead to >> performance issues. >> >> You can either put stuff in the field on your indexing client or >> create a custom update component, perhaps the simplest would be a >> "StatelessScriptUpdateProcessorFactory: >> >> see: https://cwiki.apache.org/confluence/display/solr/ >> Update+Request+Processors#UpdateRequestProcessors- >> UpdateRequestProcessorFactories >> >> Best, >> Erick >> >> On Mon, Jul 3, 2017 at 10:52 AM, Pablo Anzorena >> wrote: >> > Hey, >> > >> > I was wondering if there is some way to add fields "on the fly" based on >> > arithmetic operations on other fields. For example add a new field >> > "custom_field" = log(field1) + field2 -5. >> > >> > Thanks. >>
Re: Solr dynamic "on the fly fields"
Thanks Erick, For my use case it's not possible any of those solutions. I have a multitenancy scheme in the most basic level, that is I have a single collection with fields (clientId, field1, field2, ..., field50) attending many clients. Clients can create custom fields based on arithmetic operations of any other field. So, is it possible to update let's say field49 with the follow operation: log(field39) + field25 on clientId=43? Do field39 and field25 need to be stored to accomplish this? Is there any other way to avoid storing them? Thanks! 2017-07-03 15:00 GMT-03:00 Erick Erickson: > There are two ways: > 1> define a dynamic field pattern, i.e. > > > > Now just add any field in the doc you want. If it ends in "_sum" and > no other explicit field matches you have a new field. > > 2> Use the managed schema to add these on the fly. I don't recommend > this from what I know of your use case, this is primarily intended for > front-ends to be able to modify the schema and/or "field guessing". > > I do caution you though that either way don't go over-the-top. If > you're thinking of thousands of different fields that can lead to > performance issues. > > You can either put stuff in the field on your indexing client or > create a custom update component, perhaps the simplest would be a > "StatelessScriptUpdateProcessorFactory: > > see: https://cwiki.apache.org/confluence/display/solr/ > Update+Request+Processors#UpdateRequestProcessors- > UpdateRequestProcessorFactories > > Best, > Erick > > On Mon, Jul 3, 2017 at 10:52 AM, Pablo Anzorena > wrote: > > Hey, > > > > I was wondering if there is some way to add fields "on the fly" based on > > arithmetic operations on other fields. For example add a new field > > "custom_field" = log(field1) + field2 -5. > > > > Thanks. >
Re: Solr dynamic "on the fly fields"
There are two ways: 1> define a dynamic field pattern, i.e. Now just add any field in the doc you want. If it ends in "_sum" and no other explicit field matches you have a new field. 2> Use the managed schema to add these on the fly. I don't recommend this from what I know of your use case, this is primarily intended for front-ends to be able to modify the schema and/or "field guessing". I do caution you though that either way don't go over-the-top. If you're thinking of thousands of different fields that can lead to performance issues. You can either put stuff in the field on your indexing client or create a custom update component, perhaps the simplest would be a "StatelessScriptUpdateProcessorFactory: see: https://cwiki.apache.org/confluence/display/solr/Update+Request+Processors#UpdateRequestProcessors-UpdateRequestProcessorFactories Best, Erick On Mon, Jul 3, 2017 at 10:52 AM, Pablo Anzorenawrote: > Hey, > > I was wondering if there is some way to add fields "on the fly" based on > arithmetic operations on other fields. For example add a new field > "custom_field" = log(field1) + field2 -5. > > Thanks.
Solr dynamic "on the fly fields"
Hey, I was wondering if there is some way to add fields "on the fly" based on arithmetic operations on other fields. For example add a new field "custom_field" = log(field1) + field2 -5. Thanks.