Are you doing a commit after every document? Is the index on local disk? That is very slow indexing. With four shards and smaller documents, we can index about a million documents per minute.
wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Jul 19, 2018, at 1:28 AM, Emir Arnautović <emir.arnauto...@sematext.com> > wrote: > > Hi Francois, > If I got your numbers right, you are indexing on a single server and indexing > rate is ~31 doc/s. I would first check if something is wrong with indexing > logic. You check where the bottleneck is: do you read documents from DB fast > enough, do you batch documents… > Assuming you cannot have better rate than 30 doc/s and that bottleneck is > Solr, in order to finish it in 6h, you need to parallelise indexing on Solr > by splitting index to ~6 servers and have overall indexing rate of 180 doc/s. > > Thanks, > Emir > -- > Monitoring - Log Management - Alerting - Anomaly Detection > Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > > > >> On 19 Jul 2018, at 09:59, servus01 <francois.rein...@sportcast.de> wrote: >> >> Would like to ask what your recommendations are for a new performant Solr >> architecture. >> >> SQL DB 4M documents with up to 5000 metadata fields each document [2xXeon >> 2.1Ghz, 32GB RAM] >> Actual Solr: 1 Core version 4.6, 3.8M documents, schema has 300 metadata >> fields to import, size 3.6GB [2xXeon 2.4Ghz, 32GB RAM] >> (atm we need 35h to build the index and about 24h for a mass update which >> affects the production) >> >> Building the index should be less than 6h. Sometimes we change some of the >> Metadata fields which affects most of the documents and therefore a >> massupdate / reindex is necessary. Reindex is ok also for about 6h (night) >> but should not have an impact to user queries. Anyway, every faster indexing >> is very welcome. We will have max. 20 - 30 CCUser. >> >> So i asked myself. How many nodes, Lshards, replicas ect. Could someone >> please give me recommendation for a fast working architecture. >> >> really appreciate this, best >> >> Francois >> >> >> >> >> >> >> >> >> >> -- >> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html >
