On Thu, Jul 20, 2017 at 3:05 PM, Chaitanya Sabbineni <sabbinen...@gmail.com> wrote: > Hi, > > just to explain you what exactly is happening. We are taking back up of the > server at 12:00 AM daily.which is a type of offline backup.After backup > server is running fine for 2 hours and exactly at 2:00am server is stopping > on its own. This issue of stopping is not occurring daily.And when ever > this issue occur it is at 2:00 AM.
Always stopping at 2:00 AM is very suspicious. As you're on linux and I've seen user experience similar things before before, please verify that you don't have a cronjob that's killing/restarting the service in addition to the previous suggestion to check for logrotate. You said it doesn't stop daily, but does it stop weekly or every N days? I'm sure that there's some sort of pattern to this in addition to stopping at a specific time. > In order to check more logs we tried to increase the logging level from > Fine to Finest and upon changing of the logging level ,server is not > stopping on its own. > > when I listed the top command in Linux which will return all the running or > active process it's not listing tomcat which means its stopped. > > I checked the available memory at that time and its 139mb. > > From the error log can anyone help in pointing to the timer task that is > preventing the jvm to stop. > > Thanks in advance > > On Thu, 20 Jul 2017 11:37 pm Mark Eggers, <its_toas...@yahoo.com.invalid> > wrote: > >> Chaitanya, >> >> This will be long and somewhat speculative. >> >> On 7/20/2017 9:00 AM, Christopher Schultz wrote: >> > Chaitanya, >> > >> > On 7/20/17 11:03 AM, Chaitanya Sabbineni wrote: >> >> Stop script in the sense it's Catalina script only but we usually >> >> stop tomcat using the command Catalina.sh stop. But in our case we >> >> are not manually executing this script to stop tomcat and tomcat is >> >> stopping on its own. >> > >> >> our main problem here us tomcat is stopping on its own and it needs >> >> a restart. >> > >> > Right. >> > >> >> If I understand you correct you are telling TimerThread that does >> >> not stop when the application is shut down. Can you let me know >> >> what actually the timer thread mean. And moreover if the timer >> >> thread didn't stop ideally tomcat shouldn't stop but in our case >> >> it's stopping. >> > >> > Tomcat is stopping but the JVM is not. If your application were to >> > shut-down cleanly, then the JVM would exit as well. This is unrelated >> > to your real problem (unexpected Tomcat shutdown), but you might want >> > to look into fixing that, because it makes your application impossible >> > to reload without risking serious heap space problems. >> > >> >> Yes my question is why Tomcat is being shut down at all. >> > >> >> Yes when ever tomcat is stopping on own(not daily) it stops at >> >> 02:00 . You mentioned that your guess is that we are using a >> >> service runner that is configured to bounce your services at >> >> 02:00.Can let me know what this service runner is and how to check >> >> it. >> > >> > I know nothing about your environment. Until you mentioned >> > "catalina.sh stop" above, I didn't even know you were on a UNIX-like >> > environment. Honestly, I assumed you were on Windows because >> > "mysterious service stoppage" has Microsoft Windows behavior written >> > all over it. >> > >> > There are two ways to trigger a Tomcat shut down: >> > >> > 1. Send a TERM signal to the process >> > 2. Connect to Tomcat's shutdown listener (default: port 8005) and give >> > the shutdown command (default: "SHUTDOWN") >> > >> > You can eliminate one of those possibilities by setting the shutdown >> > port in server.xml to "" (empty) which will disable this type of shutdow >> > n: >> > >> > <Server port="" ... >> > >> > You cannot disable the other type of shutdown... any user on the >> > system who can send a TERM signal to your process could terminate Tomcat >> > . >> > >> > As for catching whoever is shutting down your Tomcat, you may want to >> > look at who has administrative access to your server, and who has >> > access to the user running your Tomcat server. >> > >> > Check your syslog to find sudo and cron events that might be >> > automatically shutting-down Tomcat. >> > >> > If you want to catch a TCP connection, you will likely have to enable >> > tcpwrappers, iptables, ipfw, etc. to log connections to port 8005. >> > Those logs will only tell you that the command is being sent, not who >> > is sending it. >> > >> > -chris >> >> I am going to go out on a limb here and try to explain things. Please >> note that this is all based upon reading between the lines, and may not >> at all reflect what is actually going on. >> >> Overview >> -------- >> >> I suspect the following: >> >> 1. Logrotate of catalina.out at 2 AM >> 2. Tomcat JVM fails to exit, then restart >> >> Detail >> ------ >> >> 1. Logrotate (or other log rotation utility) >> >> There are several ways that one can use to rotate catalina.out. See the >> following: >> >> https://wiki.apache.org/tomcat/FAQ/Logging#Q10 >> >> Some system admins actually stop Tomcat, rotate the logs, and then start >> Tomcat. This has the advantage over the logrotate's copytruncate option >> in that there is no possibility of partial log entries. >> >> 2. JVM fails to exit >> >> From your error log, you have a TimeerTask thread that is not shutting >> down. This prevents the JVM from exiting (see Chris's comments). I >> suspect that this then prevents the start script from starting Tomcat >> again (depending on the script). >> >> Solutions >> --------- >> >> 1. Fix your application so that the Tomcat JVM exits cleanly >> >> Use a thread pool. Manage the thread pool in a servlet context listener >> (creation, destruction). >> >> This should be done in addition to anything else. >> >> 2. Talk to your system admin to see if log rotation is being used >> >> Use copytruncate with logrotate rather than stopping and starting the >> Tomca service. >> >> This is assuming that you're using logrotate, and that there is a >> logrotate process that kicks off every morning at 2 AM. >> >> 3. Use another method for rotating catalina.out >> >> There are other methods for rotating catalina.out mentioned in the link >> above. >> >> Again, this is assuming that your system admin has implemented some log >> rotation which is causing the problem. >> >> 4. See Chris's comments above concerning potential security issues >> >> Finally >> ------- >> >> Your catalina.out file should be small, and consist of startup / >> shutdown messages from Tomcat. Other (application) information should go >> into application-specific log files. This means that you should >> implement some sort of logging for your applications. >> >> In other words, there should be little need to periodically rotate >> catalina.out. >> >> . . . just my two cents >> /mde/ >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
