We are using Solr 5.4 to index TBs of documents in a bulk fashion to get the cluster up and running. Indexing is over HTTP round robin as directed by zookeeper.
Each of the 13 nodes is receiving about 6-8 MB/s on the NIC but solr is writing around 20 to 25 thousand times per second (4k block size). My question is what is Solr doing writing all this data to disk (80-100MB/s)? Over a three hour run with 4.5 million docs we only committed 20 some times but disk activity was pretty constant at the above levels. Is there more going on than tlogs, commits and merges? When we moved from 1 minute autoCommit to 10 we committed less per the log messages but I expected the bigger initial segments to result in less merging thus lower disk activity. But testing showed no significant change in disk writing. Thanks for any help. Andrew
