mikemccand commented on issue #16392: URL: https://github.com/apache/lucene/issues/16392#issuecomment-5178917838
I suspect most codec formats could trivially track per-field start/end? Most our our codec impls concatenate fields sequentially, so the reader is something like "for field in fields readVLong() or so for the start of that field", and keeping track of the ends of each field (= start of next field, or EOF, roughly) seems trivial. I think we can design for this common case? Hard cases like stored fields and term vectors are already problematic (regardless of SAX (visitor) vs DOM (fully materialized objects) API), but maybe as a visitor API it might invoke something like "docs 0-9 took 22 KB", "docs 10-19 took 1233 KB", etc. This is a nice enabler of the visitor API: you can call with much finer detail than we'd want to enumerate in a full DOM (XML reference) style approach. But I would skip the stored fields on this first cut ... just tackle visitor approach for the codec components that do clearly break into per-field? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
