[
https://issues.apache.org/jira/browse/SOLR-9579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530812#comment-15530812
]
Mikhail Khludnev commented on SOLR-9579:
----------------------------------------
yep.. hadoop API is known for such optimization, and Lucene's fields are
designed to be reused. But what is a wall clock gain? Java is already optimized
for such memory waste, attempts for fixing might even make things worse.
> Reuse lucene FieldType in createField flow during ingestion
> -----------------------------------------------------------
>
> Key: SOLR-9579
> URL: https://issues.apache.org/jira/browse/SOLR-9579
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 6.x, master (7.0)
> Environment: This has been primarily tested on Windows 8 and Windows
> Server 2012 R2
> Reporter: John Call
> Priority: Minor
> Labels: gc, memory, reuse
> Fix For: 6.x, master (7.0)
>
> Attachments: SOLR-9579.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> During ingestion createField in FieldType is being called for each field on
> each document. For the subclasses of FieldType without their own
> implementation of createField the lucene version of FieldType is created to
> be stored along with the value. However the lucene FieldType object is
> identical when created from the same SchemaField. In testing ingestion of one
> million rows with 22 field each we were creating 22 million lucene FieldType
> objects when only 22 are needed. Solr should lazily initialize a lucene
> FieldType for each SchemaField and reuse them for future ingestion. Not only
> does this relieve memory usage but also relieves significant pressure on the
> gc.
> There are also subclasses of Solr FieldType which create separate Lucene
> FieldType for stored fields instead of reusing the static in StoredField.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
