It's safe to say that you are on thin ice - try it and if it works well for
your specific data and requirements, great. But if it eventually fails, you
can't say we didn't warn you!
Generally, I would say that "a few hundred" fields is the recommended
limit - not that more will definitely cause problems, but because you will
be beyond common usage and increasingly sensitive to amount of data and
Java/JVM performance capabilities.
-- Jack Krupansky
-----Original Message-----
From: Mike L.
Sent: Saturday, February 8, 2014 2:12 PM
To: solr-user@lucene.apache.org
Cc: solr-user@lucene.apache.org
Subject: Re: Max Limit to Schema Fields - Solr 4.X
That was the original plan.
However its important to preserve the originating field that loaded the
value. The data is very fine and granular and each field stores a particular
value. When searching the data against solr - it would be important to know
what docs contain that particular data from that particular field.
(fielda:value, fieldb:value ) where as searching field:value would hide what
originating field loaded this value.
Im going to try loading all 3000 fields in the schema and see how that goes.
Only concern is doing boolean searches and whether or not Ill run into URL
length issues but I guess Ill find out soon.
Thanks again!
Sent from my iPhone
On Feb 6, 2014, at 1:02 PM, Erick Erickson <erickerick...@gmail.com>
wrote:
Sometimes you can spoof the many fields problem by using prefixes on the
data. Rather than fielda, fieldb... Have field and index values like
fielda_value, fieldb_value into a single field. Then do the right thing
when searching. Watch tokenization though.
Best
Erick
On Feb 5, 2014 4:59 AM, "Mike L." <javaone...@yahoo.com> wrote:
Thanks Shawn. This is good to know.
Sent from my iPhone
On Feb 5, 2014, at 12:53 AM, Shawn Heisey <s...@elyograg.org> wrote:
On 2/4/2014 8:00 PM, Mike L. wrote:
I'm just wondering here if there is any defined limit to how many
fields can be created within a schema? I'm sure the configuration
maintenance of a schema like this would be a nightmare, but would like to
know if its at all possible in the first place before It may be
attempted.
There are no hard limits on the number of fields, whether they are
dynamically defined or not. Several thousand fields should be no
problem. If you have enough system resources and you don't run into an
unlikely bug, there's no reason it won't work. As you've already been
told, there are potential performance concerns. Depending on the exact
nature of your queries, you might need to increase maxBooleanClauses.
The only hard limitation that Lucene really has (and by extension, Solr
also has that limitation) is that a single index cannot have more than
about two billion documents in it - the inherent limitation on a Java
"int" type. Solr can use indexes larger than this through sharding.
See the very end of this page:
https://lucene.apache.org/core/4_6_0/core/org/apache/lucene/codecs/lucene46/package-summary.html#Limitations
Thanks,
Shawn
