I added distribUpdateConnTimeout and distribUpdateSoTimeout to solr.xml and the commit did timeout.(btw, is there any way to view solr.xml in the admin console?).
Also, although we do have an init.d start/stop script for Solr, the 'stop' command was not executed during shutdown because there was no lock file for the script in '/var/lock/subsys'. I didn't know about this until I google'd around and found ' http://www.redhat.com/magazine/008jun05/departments/tips_tricks'. When I added the lock file, both the AWS 'stop' and 'terminate' actions did result in an orderly shutdown of the replica which caused the Solr-leader to get an exception and update the live_nodes, gracefully. So now, the timeouts should only play a backup role. Thanks for the help, Peter On Fri, Dec 12, 2014 at 5:21 PM, Chris Hostetter <hossman_luc...@fucit.org> wrote: > > > : No, I wasn't aware of these. I will give that a try. If I stop the Solr > : jetty service manually, things recover fine, but the hang occurs when I > : 'stop' or 'terminate' the EC2 instance. The Zookeeper leader reports a > > I don't know squat about AWS Auto-Scaling, (and barely anything about AWS) > but what you describe makes it sound like maybe your "machine" (ie AMI?) > isn't really configured very well? > > Do you have some init.d/systemd type scripts to ensure a clean shutdown of > Solr when the machine is shutdown/rebooted? That seems like a pretty good > idea in general (in dependent of wether you are using Auto-Scaling) and -- > assuming AWS auto-scaling does clean OS shutdowns when terminating > instances -- would probably solve your problem. It would help ensure you > would never have to wait on the timeouts -- the nodes will each explicitly > tell ZK they are going bye-bye. > > if you do have things setup so that *manually* shutting down your > instances executes a clean shutdown of solr, but AWS Auto-Scaling is > actaully totally brutal and doesn't even do a clean shutdown of your > virtual machines -- just yanks the virtual power cord -- perhaps you could > implement one of these "LifecycleHook" options that poped up when i did > some googling for "AWS Auto-Scale termination" to explicitly do a clean > shutdown of the Solr process" before the machine vanishes into thin air? > > > > -Hoss > http://www.lucidworks.com/ >
