> * Do NOT commit after each batch of 1000 docs. Instead, commit as seldom as your requirements allows, e.g. try commitWithin=60000 to commit every minute
this is the big one. commit after the entire process is done or on a timer, if you don't need NRT searching, rarely does anyone ever need that. the commit is a heavy operation and takes about the same time if you are committing 1000 documents or 100k documents. On Wed, Jun 8, 2022 at 10:40 AM Jan Høydahl <[email protected]> wrote: > * Go multi threaded for each core as Shawn says. Try e.g. 2, 3 and 4 > threads > * Experiment with different batch sizes, e.g. try 500 and 2000 - depends > on your docs what is optimal > * Do NOT commit after each batch of 1000 docs. Instead, commit as seldom > as your requirements allows, e.g. try commitWithin=60000 to commit every > minute > > Tip: Try to push Solr metrics to DataDog or some other service, where you > can see a dashboard with stats on requests/sec, RAM, CPU, threads, GC etc > which may answer your last question. > > Jan > > > 8. jun. 2022 kl. 14:06 skrev Shawn Heisey <[email protected]>: > > > > On 6/8/2022 3:35 AM, Marius Grigaitis wrote: > >> * 9 different cores. Each weighs around ~100 MB on disk and has > >> approximately 90k documents inside each. > >> * Updating is performed using update method in batches of 1000, around 9 > >> processes in parallel (split by core) > > > > This means that indexing within each Solr core is single-threaded. The > way to increase indexing speed is to index in parallel with multiple > threads or processes per index. If you can increase the CPU power > available on the Solr server when you increase the number of > processes/threads sending data to Solr, that might help. > > > > Thanks, > > Shawn > > > >
