Yeah that sounds as if it would be too expensive. I wasn't quite sure what would be involved..
On Wed, Nov 18, 2020 at 3:56 PM Michael Froh <[email protected]> wrote: > > I didn't try creating a new IndexWriter for each batch, but I was assuming > that would be heavier, as it would allocate a new DocumentsWriter, and > through that new DocumentsWriterPerThreads. Skimming through the code for > DWPT, it looks like there are various pools involved in creating each DWPT's > instance of DefaultIndexingChain, which might be expensive to create > frequently, rather than reusing on flush(). > > Also I was partly motivated by laziness. The production code I'm borrowing > for this prototype doesn't make it easy to recreate the IndexWriterConfig, > and IWC is not reusable across IndexWriter instances. > > On Wed, Nov 18, 2020 at 12:25 PM Michael Sokolov <[email protected]> wrote: >> >> I'm curious if you tried creating a new IndexWriter for each batch? >> >> On Wed, Nov 18, 2020 at 1:18 PM Michael Froh <[email protected]> wrote: >> > >> > I have some code that is kind of abusing IndexWriter.deleteAll(). In >> > short, I'm basically experimenting with using tiny (one block of joined >> > parent/child documents) indexes as a serialized format to index on one >> > fleet and then merge these tiny indexes on another fleet. I'm doing this >> > by indexing a block, committing, storing the contents of the index >> > directory in a zip file, invoking deleteAll(), and repeating. Believe it >> > or not, the performance is not terrible. (Currently getting about 20% of >> > the throughput I see with regular indexing.) >> > >> > Regardless of my serialization shenanigans above, I've found that >> > performance degrades over time for the process, as it spends more time >> > allocating and freeing memory. Analyzing some heap dumps, it's because >> > FieldInfos.byNumber is getting bigger and bigger. IndexWriter.deleteAll() >> > doesn't truly reset state. Specifically, it calls >> > globalFieldNumberMap.clear(), which clears all of the FieldNumbers >> > collections, but it doesn't reset lowestUnassignedFieldNumber. So, that >> > number keeps counting up, and new instances of FieldInfos allocate larger >> > and larger arrays (and only use the top indices). >> > >> > Has anyone else encountered this? Can I open an issue for resetting >> > lowestUnassignedFieldNumber in FieldNumbers.clear()? Is there any risk in >> > doing so? >> > >> > (For my specific use-case, I would be okay with not clearing >> > globalFieldNumberMap at all, since the set of fields is bounded, but >> > assigning new field numbers is probably among the least of my costs.) >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
