Do you have any close count of how many max dynamic fields you may have (1k, 2k or 3k etc.). In one of our index we have a total around 2K dynamic fields across all documents.
My suggestion would be to try out dynamic fields for the use case you are describing and do some real performance test. Thanks, Susheel On Tue, Jun 27, 2017 at 3:01 PM, Saurabh Sethi <saurabh.se...@sendgrid.com> wrote: > We have key-value pairs that need to be searchable. We are looking for best > approach, both in terms of indexing (fast as well as space efficient) as > well as retrieval (fast search). > > Right now, the two approaches that we have are: Nested docs or dynamic > fields (myField_*_time:some date) > > The number of dynamic fields would definitely be > 1k. > > We wanted to get an idea which of these approaches work best or if there a > third approach which is better than nested and dynamic fields. > > On Tue, Jun 27, 2017 at 5:39 AM, Susheel Kumar <susheel2...@gmail.com> > wrote: > > > Can you describe your use case in terms of what business functionality > you > > are looking to achieve. > > > > Thanks, > > Susheel > > > > On Mon, Jun 26, 2017 at 4:26 PM, Saurabh Sethi < > saurabh.se...@sendgrid.com > > > > > wrote: > > > > > Number of dynamic fields will be in thousands (millions of users + > > > thousands of events shared between subsets of users). > > > > > > We also thought about indexing in one field with value being > > > fieldname_fieldvalue. Since we support range queries for dates and > > numbers, > > > it won't work out of box. > > > > > > On Mon, Jun 26, 2017 at 1:05 PM, Erick Erickson < > erickerick...@gmail.com > > > > > > wrote: > > > > > > > How many distinct fields do you expect across _all_ documents? That > > > > is, if doc1 has 10 dynamic fields and doc2 has 10 dynamic fields, > will > > > > there be exactly 10 fields total or more than 10 when you consider > > > > both documents? > > > > > > > > 100s of fields total across all documents is a tractable problem. > > > > thousands of dynamic fields total is going to be a problem. > > > > > > > > One technique that people do use is to index one field with a prefix > > > > rather than N dynamic fields. So you have something like > > > > dyn1_val1 > > > > dyn1_val2 > > > > dyn4_val67 > > > > > > > > Only really works with string fields of course. > > > > > > > > Best, > > > > Erick > > > > > > > > On Mon, Jun 26, 2017 at 10:11 AM, Saurabh Sethi > > > > <saurabh.se...@sendgrid.com> wrote: > > > > > We have two requirements: > > > > > > > > > > 1. Indexing and storing event id and its timestamp. > > > > > 2. Indexing and storing custom field name and value. The fields can > > be > > > of > > > > > any type, but for now lets say they are of types string, date and > > > number. > > > > > > > > > > The events and custom fields for any solr document can easily be in > > > > > hundreds. > > > > > > > > > > We are looking at two different approaches to handle these > scenarios: > > > > > > > > > > 1. *Dynamic fields* - Have the fields name start with a particular > > > > pattern > > > > > like for string, the pattern could be like str_* and for event > could > > be > > > > > eventid_* > > > > > 2. *Parent/child fields* - This seems to be an overkill for our use > > > case > > > > > since it's more for hierarchical data. Also, the parent and all its > > > > > children need to be reindexed on update which defeats the purpose - > > we > > > > are > > > > > now reindexing multiple docs instead of one with dynamic fields. > But > > it > > > > > allows us to store custom field name along with its value unlike > > > dynamic > > > > > fields where we will have to map user supplied custom field to some > > > other > > > > > name based on type. > > > > > > > > > > Has anyone handled similar scenarios with Solr? If so, which > approach > > > > would > > > > > you recommend based on your experience? > > > > > > > > > > We are using solr 6.6 > > > > > > > > > > Thanks, > > > > > Saurabh > > > > > > > > > > > > > > > > -- > > > Saurabh Sethi > > > Principal Engineer I | Engineering > > > > > > > > > -- > Saurabh Sethi > Principal Engineer I | Engineering >
