Hi, Sometimes after a full poweroff of the solr cloud nodes, we see missing documents from the index. Is there anything about our setup or our recovery procedure that could cause this? Details are below:
We see the following (somewhat random) behaviour: - add 10 documents to index. Commit. - query for all documents - 10 documents returned. - restart all solr nodes and reset the collection (procedure is below). - query for all documents 10 documents returned. - restart+reset all again. - sometimes 7, 8, 9, or 10 documents returned. To summarize, after a full reboot of all the solr nodes, we are finding that (sometimes) not all documents are in the index. This situation doesn't remedy itself by waiting. Restarting all will sometimes re-add them, sometimes not. Our procedure for recovering from a hard poweroff is: - manually delete all *.lock files from the index folders on hdfs. - fully delete the znode from zookeeper. - re-add an empty znode in zookeeper. - start up all solr nodes. - re-add the configset. - re-issue the collection create command. After doing the above, we find that we are able to see all of the files in the index about 60% of the time. Other times, we are missing some documents. Some other things about our environment: - we're doing this test with 1 collection that has 18 shards distributed across 3 solr cloud nodes. - solr version 7.5.0 - hdfs is not running on the solr nodes, and is not being restarted. Any thoughts or tips are greatly appreciated, Kyle -- CONFIDENTIALITY NOTICE: The information contained in this email is privileged and confidential and intended only for the use of the individual or entity to whom it is addressed. If you receive this message in error, please notify the sender immediately at 613-729-1100 and destroy the original message and all copies. Thank you.
