Thanks Erick and Shawn , Thank you for your patience. I said that the above phenomenon was caused by the IO, cpu, memory, and network io. The swap was turned off and the machine's memory was sufficient. When the speed of indexing is declining, QTime is found to take 3 seconds to 4 seconds to reload the index, so it can be guessed that it is more likely to be a Solr problem than a jetty. It is worth mentioning that when the speed of the index under construction dropped sharply, the Solr used only about 5% of the CPU, and when it was normal, the CPU usage was 200 percent, and the overall system's CPU usage was 100 percent. About twenty.
Basic information: 1) The data volume of each collection is between 2 billion and 3 billion. 2) The configuration of the machine is 24 cpu and 128G memory. 3) The disk usage per copy is about 10G. In addition, I noticed that the work of zookeeper is normal and there is no error or warning message. So all these phenomena make me think that the internal specific mechanism of solr may lead to a sharp drop in the index construction speed. At present, it seems that our solr's machine resources are sufficient. As for the reduction of the number of collections that you said, we also have this plan, and we are looking for ways to reform it. Are there any other suggestions? Best . miaohq 2018-03-11 10:15 GMT+08:00 spoonerk <john.spoo...@gmail.com>: > Wow thanks. Just trying to unsubscribe. Most email lists let u do that > > On Mar 10, 2018 2:36 PM, "Erick Erickson" <erickerick...@gmail.com> wrote: > > > Spoonerk: > > > > You say you've tried "many times", but you haven't provided full > > header as described in the "problems" link at the link below. You > > haven't e-mailed the list owner as suggested in the "problems" link. > > You haven't, in short, provided any of the information that's > > necessary to actually unsubscribe you. > > > > Please follow the instructions here: > > http://lucene.apache.org/solr/community.html#mailing-lists-irc. In > > particular look at the "problems" link. > > > > You must use the _exact_ same e-mail as you used to subscribe. > > > > If the initial try doesn't work and following the suggestions at the > > "problems" link doesn't work for you, let us know. But note you need > > to show us the _entire_ return header to allow anyone to diagnose the > > problem. > > > > Best, > > Erick > > > > On Sat, Mar 10, 2018 at 1:03 PM, spoonerk <john.spoo...@gmail.com> > wrote: > > > I have manually unsubscribed many times. But I still get emails from > the > > > list. Can some admin please unsubscribe me? > > > > > > On Mar 9, 2018 9:52 PM, "苗海泉" <mseaspr...@gmail.com> wrote: > > > > > >> hello,We found a problem. In solr 6.0, the indexing speed of solr is > > >> influenced by the number of solr collections. The speed is normal > before > > >> the limit is reached. If the limit is reached, the indexing speed will > > >> decrease by 50 times. > > >> > > >> In our environment, there are 49 solr nodes. If each collection has 25 > > >> shards, you can maintain high-speed indexing until the total number of > > >> collections is about 900. To reduce the number of collections to the > > limit, > > >> the speed will increase. Go up. > > >> If each collection is 49 shards, the total number of collections can > > only > > >> be about 700, exceeding this value will cause the index to drop > > >> dramatically. > > >> In the explanation, we are single copies, and multiple copies will > cause > > >> serious stability problems in the large solr cluster environment. > > >> > > >> At first I suspect that it was due to too many thread submissions, and > > >> there are still problems with this method, so I'm inclined to > > >> searcherExecutor thread pool thread. This is just my guess, I want to > > know > > >> the real reason. Can someone know if I can help? > > >> > > >> Also, I noticed that the searcherExecutor thread and solr collection's > > >> shards basically correspond to each other. How can I reduce the number > > of > > >> threads or even close it? Although there are many collections in our > > >> environment, there are few queries and it is not necessary to keep the > > >> threads open to provide queries. This is too wasteful. > > >> > > >> thank you . > > >> > > > -- ============================== 联创科技 知行如一 ==============================
