I'm curious if you tried creating a new IndexWriter for each batch? On Wed, Nov 18, 2020 at 1:18 PM Michael Froh <[email protected]> wrote: > > I have some code that is kind of abusing IndexWriter.deleteAll(). In short, > I'm basically experimenting with using tiny (one block of joined parent/child > documents) indexes as a serialized format to index on one fleet and then > merge these tiny indexes on another fleet. I'm doing this by indexing a > block, committing, storing the contents of the index directory in a zip file, > invoking deleteAll(), and repeating. Believe it or not, the performance is > not terrible. (Currently getting about 20% of the throughput I see with > regular indexing.) > > Regardless of my serialization shenanigans above, I've found that performance > degrades over time for the process, as it spends more time allocating and > freeing memory. Analyzing some heap dumps, it's because FieldInfos.byNumber > is getting bigger and bigger. IndexWriter.deleteAll() doesn't truly reset > state. Specifically, it calls globalFieldNumberMap.clear(), which clears all > of the FieldNumbers collections, but it doesn't reset > lowestUnassignedFieldNumber. So, that number keeps counting up, and new > instances of FieldInfos allocate larger and larger arrays (and only use the > top indices). > > Has anyone else encountered this? Can I open an issue for resetting > lowestUnassignedFieldNumber in FieldNumbers.clear()? Is there any risk in > doing so? > > (For my specific use-case, I would be okay with not clearing > globalFieldNumberMap at all, since the set of fields is bounded, but > assigning new field numbers is probably among the least of my costs.)
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
