Recently upgraded to 4.3.1 but this problem has persisted for a while now ...
I'm using the following configuration when starting Jetty: -XX:OnOutOfMemoryError="/home/solr/oom_killer.sh 83 %p" If an OOM is triggered during Solr web app initialization (such as by me lowering -Xmx to a value that is too low to initialize Solr with), then the script gets called and does what I expect! However, once the Solr webapp initializes and Solr is happily responding to updates and queries. When an OOM occurs in this situation, then the script doesn't actually get invoked! All I see is the following in the stdout/stderr log of my process: # # java.lang.OutOfMemoryError: Java heap space # -XX:OnOutOfMemoryError="/home/solr/oom_killer.sh 83 %p" # Executing /bin/sh -c "/home/solr/oom_killer.sh 83 21358"... The oom_killer.sh script doesn't actually get called! So to recap, it works if an OOM occurs during initialization but once Solr is running, the OOM killer doesn't fire correctly. This leads me to believe my script is fine and there's something else going wrong. Here's the oom_killer.sh script (pretty basic): #!/bin/bash SOLR_PORT=$1 SOLR_PID=$2 NOW=$(date +"%Y%m%d_%H%M") ( echo "Running OOM killer script for process $SOLR_PID for Solr on port 89$SOLR_PORT" kill -9 $SOLR_PID echo "Killed process $SOLR_PID" exec /home/solr/solr-dg/dg-solr.sh recover $SOLR_PORT & echo "Restarted Solr on 89$SOLR_PORT after OOM" ) | tee oom_killer-89$SOLR_PORT-$NOW.log Anyone see anything like this before? Suggestions on where to begin tracking down this issue? Cheers, Tim
