[
https://issues.apache.org/jira/browse/SOLR-10117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Smiley updated SOLR-10117:
--------------------------------
Attachment: SOLR_10117_large_fields.patch
Here's a patch that adds a "large" boolean property to schema fields. Such
fields, in conjunction with lazy field loading, will get their own separate
lazy document. _This is one piece of a larger puzzle._
It would be awesome to _instead_ have large-ness detected dynamically without
having to declare them as large as I'm doing here but that has some issues.
First if the lazy field loading wanted to know the size then it has to incur
the cost of loading it, which defeats the point. A possible solution I'm in
favor of (yet might be controversial?) would be Solr's DocumentBuilder
detecting large string values and then if there is one then adding the name to
a proposed docValues {{\_largeFields\_}} field. Then lazy field loading could
examine the values. An alternative variation on this is to save it as a stored
value that comes first in the stored document, since lazy field loading has to
go to disk for this any way. I actually like that better than using docValues.
It might even go further and place the largest field value(s) last to benefit
from a Lucene level optimization I got in a while back.
Notice this LargeFieldsTest uses some testing techniques I want to popularize
and refine. There are no new schema/solrconfig files despite that this test
defines fields, field types, and makes solrconfig changes. And it doesn't copy
configs to do this.
> Big docs and the DocumentCache; umbrella issue
> ----------------------------------------------
>
> Key: SOLR-10117
> URL: https://issues.apache.org/jira/browse/SOLR-10117
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: David Smiley
> Assignee: David Smiley
> Attachments: SOLR_10117_large_fields.patch
>
>
> This is an umbrella issue for improved handling of large documents (large
> stored fields), generally related to the DocumentCache or SolrIndexSearcher's
> doc() methods. Highlighting is affected as it's the primary consumer of this
> data. "Large" here is multi-megabyte, especially tens even hundreds of
> megabytes. We'd like to support such users without forcing them to choose
> between no DocumentCache (bad performance), or having one but hitting OOM due
> to massive Strings winding up in there. I've contemplated this for longer
> than I'd like to admit and it's a complicated issue with difference concerns
> to balance.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
