[
https://issues.apache.org/jira/browse/SOLR-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15085882#comment-15085882
]
Steve Rowe commented on SOLR-8495:
----------------------------------
Here are the ways I can think of to address this problem:
# Autodetect space-separated text above a (customizable? maybe 256 bytes or so
by default?) threshold as tokenized text rather than as StrField.
# Make StrField auto-truncate at Lucene's 32k limit.
# Make the guessed "strings" fieldType be TextField that uses KeywordTokenizer,
and add a token filter that truncates StrField terms to Lucene's 32k limit
I like #1 the best, because I think it aligns with likely user expectations,
and it doesn't silently throw away data.
> Schemaless mode cannot index large text fields
> ----------------------------------------------
>
> Key: SOLR-8495
> URL: https://issues.apache.org/jira/browse/SOLR-8495
> Project: Solr
> Issue Type: Bug
> Components: Data-driven Schema, Schema and Analysis
> Affects Versions: 4.10.4, 5.3.1, 5.4
> Reporter: Shalin Shekhar Mangar
> Labels: difficulty-easy, impact-medium
> Fix For: 5.5, Trunk
>
>
> The schemaless mode by default indexes all string fields into an indexed
> StrField which is limited to 32KB text. Anything larger than that leads to an
> exception during analysis.
> {code}
> Caused by: java.lang.IllegalArgumentException: Document contains at least one
> immense term in field="text" (whose UTF8 encoding is longer than the max
> length 32766)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
