This sounds like a denormalization issue. Don't be afraid <G>. Actually, I've seen from 50M 50 300M small docs on a Solr node, depending on query type, hardware, etc. So that gives you a place to start being cautious about the number of docs in your system. If your full expansion of your table numbers in that range, you might be just fine denormalizing the data.
Alternatively, there's the "pseudo join" capability to consider. I'm usually hesitant to recommend that, but Joel is committing some really interesting stuff in the join area which you might take a look at if the existing pseudo-join isn't performant enough. But I'd consider denormalizing the data as the first approach. Best, Erick On Sun, Oct 13, 2013 at 8:07 AM, David Philip <davidphilipshe...@gmail.com>wrote: > Hi Jack, for the point: "each element of the array as a solr document, with > a group field and a disease field" > Did you mean it this way: > > <doc> > "group1_grp": G1 > "disease1_d": 2, > "disease2_d": 3, > </doc> > <doc> > "group1_grp": G2 > "disease1_d": 2, > "disease2_d": 3, > "disease3_d": 1, > "disease4_d": 1, > </doc> > similar to first case: having dynamic fields for disease? > Will it be performance issue if disease field increase to millions? > > > > > > > > > > > > On Sun, Oct 13, 2013 at 9:00 AM, Jack Krupansky <j...@basetechnology.com > >wrote: > > > You may be better off indexing each element of the array as a solr > > document, with a group field and a disease field. Then you can easily and > > efficiently add new diseases. Then to query a row, you query for the > group > > field having the desired group. > > > > If possible, index the array as being sparse - no document for a disease > > if it is not present for that group. > > > > -- Jack Krupansky > > > > -----Original Message----- From: David Philip > > Sent: Saturday, October 12, 2013 9:56 PM > > To: solr-user@lucene.apache.org > > Subject: Re: Storing 2 dimension array in Solr > > > > > > Hi Erick, Yes it is. But the columns here are dynamically and very > > frequently added.They can increase upto 1 million right now. So, 1 > document > > with 1 million dynamic fields, is it fine? Or any other approach? > > > > While searching through web, I found that docValues are column oriented. > > http://searchhub.org/2013/04/**02/fun-with-docvalues-in-solr-**4-2/< > http://searchhub.org/2013/04/02/fun-with-docvalues-in-solr-4-2/> > > However, I did not understand, how to use docValues to add these > columns. > > > > What is the recommended approach? > > > > Thanks - David > > > > > > > > > > > > > > On Sun, Oct 13, 2013 at 3:33 AM, Erick Erickson <erickerick...@gmail.com > >* > > *wrote: > > > > Isn't this just indexing each row as a separate document > >> with a suitable ID "groupN" in your example? > >> > >> > >> On Sat, Oct 12, 2013 at 2:43 PM, David Philip > >> <davidphilipshe...@gmail.com>**wrote: > >> > >> > Hi Erick, > >> > > >> > We have set of groups as represented below. New columns (diseases > as > >> in > >> > below matrix) keep coming and we need to add them as new column. To > that > >> > column, we have values such as 1 or 2 or 3 or 4 (exist, slight, na, > >> > notfound) for respective groups. > >> > > >> > While querying we need to get the entire row for group:"group1". We > >> will > >> > not be searching on columns(*_disease) values, index=false but stored > is > >> > true. > >> > > >> > for ex: we use, get group:"group1" and we need to get the entire row- > >> > exist,slight, not found. Hoping this explanation is clearer. > >> > > >> > disease1 disease2 disease3 > >> > group1 exist slight not found > >> > groups2 slight not found exist > >> > group3 slight exist > >> > groupK - na exist > >> > > >> > > >> > > >> > Thanks - David > >> > > >> > > >> > > >> > > >> > > >> > On Sat, Oct 12, 2013 at 11:39 PM, Erick Erickson < > >> erickerick...@gmail.com > >> > >wrote: > >> > > >> > > David: > >> > > > >> > > This feels like it may be an XY problem. _Why_ do you > >> > > want to store a 2-dimensional array and what > >> > > do you want to do with it? Maybe there are better > >> > > approaches. > >> > > > >> > > Best > >> > > Erick > >> > > > >> > > > >> > > On Sat, Oct 12, 2013 at 2:07 AM, David Philip > >> > > <davidphilipshe...@gmail.com>**wrote: > >> > > > >> > > > Hi, > >> > > > > >> > > > I have a 2 dimension array and want it to be persisted in solr. > > > >> > > How > >> > > can I > >> > > > do that? > >> > > > > >> > > > Sample case: > >> > > > > >> > > > disease1 disease2 disease3 > >> > > > group1 exist slight not found > >> > > > groups2 slight not found exist > >> > > > group2 slight exist > >> > > > > >> > > > exist-1 not found - 2 slight-3 .. can be stored like this also. > >> > > > > >> > > > Note: This array has frequent updates. Every time new disease > get's > >> > > added > >> > > > and I have to add description about that disease to all groups. > And > >> at > >> > > > query time, I will do get by row - get by group only group = > group2 > >> > row. > >> > > > > >> > > > Any suggestion on how I can achieve this? I am thankful to the > > > > >> > forum > >> > for > >> > > > replying with patience, on achieving this, i will blog and will > > > > >> > share > >> > it > >> > > > with all. > >> > > > > >> > > > Thanks - David > >> > > > > >> > > > >> > > >> > >> > > >