bq: Is my understanding about stored fields correct, that even if excluded from fl, the data on the disk for a given field would still be read as part of decompression..
Assuming any stored field (NOT docvalues) was read then this is, indeed, correct. To be pedantic about it, enough 16K blocks will be read/decompressed to get all the fields of the doc, then the necessary fields will be extracted. I gather it's kind hard to index into a compressed blob and extract just a specific field. Alexandre: I think Shawn's looking for the opposite. If useDocValuesAsStored="true" do _not_ fetch any fields where docValues=false and stored=true for fl=*. Best, Erick On Fri, Jan 13, 2017 at 6:03 PM, Shawn Heisey <apa...@elyograg.org> wrote: > On 1/13/2017 1:02 PM, Erick Erickson wrote: >> What about using the defaults in requestHandlers along with SOLR-3191 >> to accomplish this? Let's say that there was an fl-exclusion >> parameter. Now you'd be able to define an exclusion default that would >> exclude your field(s) unless overridden in your request handler. This >> could be either a default or invariant depending on how strictly you >> wanted to enforce not being able to retrieve the field. > > If it's done with a parameter, I would want the parameter to work > correctly if included multiple times, then add an exclusion default to > the appends section rather than defaults or invariants. > >> And one thing about your notion. docValues are only primitive types, >> i.e. string in this case. There's a limit I believe on how big these >> can be, 32K? Which seems rather restrictive in this case so we're back >> to stored. > > Oh, fun. 32K might be enough for my index, but it is not enough for > general usage. > > Is my understanding about stored fields correct, that even if excluded > from fl, the data on the disk for a given field would still be read as > part of decompression? That's what I was hoping to avoid by using > docValues. Just how much pain would be involved in implementing an > option to disable stored field compression, and if that became possible, > would it avoid the need to read field data that isn't used? > > Thanks, > Shawn >
