[
https://issues.apache.org/jira/browse/SOLR-7971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715540#comment-14715540
]
Yonik Seeley commented on SOLR-7971:
------------------------------------
Making 3 copies really feels heavyweight (the current code makes a single copy
in the case of large strings).
string->scratch, scratch->off-heap-buffer, off-heap-buffer->scratch,
scratch->write
And this doesn't even use less system memory... just less heap memory.
I wonder how it would do against Mikhail's idea of just calculating the UTF8
length first.
If we keep with DirectByteBuffer, we should at least eliminate one of the
copies by encoding directly to it?
> Reduce memory allocated by JavaBinCodec to encode large strings
> ---------------------------------------------------------------
>
> Key: SOLR-7971
> URL: https://issues.apache.org/jira/browse/SOLR-7971
> Project: Solr
> Issue Type: Sub-task
> Components: Response Writers, SolrCloud
> Reporter: Shalin Shekhar Mangar
> Assignee: Shalin Shekhar Mangar
> Priority: Minor
> Fix For: Trunk, 5.4
>
> Attachments: SOLR-7971-directbuffer.patch,
> SOLR-7971-directbuffer.patch, SOLR-7971.patch
>
>
> As discussed in SOLR-7927, we can reduce the buffer memory allocated by
> JavaBinCodec while writing large strings.
> https://issues.apache.org/jira/browse/SOLR-7927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700420#comment-14700420
> {quote}
> The maximum Unicode code point (as of Unicode 8 anyway) is U+10FFFF
> ([http://www.unicode.org/glossary/#code_point]). This is encoded in UTF-16
> as surrogate pair {{\uDBFF\uDFFF}}, which takes up two Java chars, and is
> represented in UTF-8 as the 4-byte sequence {{F4 8F BF BF}}. This is likely
> where the mistaken 4-bytes-per-Java-char formulation came from: the maximum
> number of UTF-8 bytes required to represent a Unicode *code point* is 4.
> The maximum Java char is {{\uFFFF}}, which is represented in UTF-8 as the
> 3-byte sequence {{EF BF BF}}.
> So I think it's safe to switch to using 3 bytes per Java char (the unit of
> measurement returned by {{String.length()}}), like
> {{CompressingStoredFieldsWriter.writeField()}} does.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
