janhoy commented on PR #4910:
URL: https://github.com/apache/solr/pull/4910#issuecomment-5680302181
Wrt heap use, the current impl requires about 5x, i.e. a 10Mb document will
require about 50-60Mb heap to keep the line and copies.
I researched some more (with AI), and there is a way we can tap into the
existing Noggit-provided streaming parser, also for ND-JSON:
* `JsonLoader` is made more extensible by exposing new methods for
`ContentStreamLoader createLoader()`, and `JSONParser createParser()`. -10/+20
lines.
* `NDJsonLoader` subclass `JsonParser` and provides overrides of those two:
* A new `NDJsonParser extends org.noggit.JSONParser`, enforcing ND-JSON
format such as strictly doc-per-line, and well-formed json per line
* An inner `SingleThreadedNDJsonLoader` enforcing the ND-JSON contract
* Inherits the logic for `split`, `f` flattening etc
Memory footprint will be the fixed 8192-char Noggit buffer, plus the largest
single field value. Plus the size of the constructed `SolrInputDocument` - but
that cost comes on top for any loader.
I agree that the footprint of the naïve approach is less than ideal, so
happy to push the described improvement if you wish.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
