Hello Thomson,
> -----Ursprüngliche Nachricht-----
> Von: Thomson Waghmare <thomsonwaghm...@gmail.com>
> Gesendet: Dienstag, 6. Juni 2023 07:44
> An: users@tomcat.apache.org
> Betreff: Tomcat web server restarts automatically
>
> Hi Team,
>
> We have Tomcat web server as a service that has 4 web applications
> deployed on it. One of them is ALARMCFG, an alarm reporting application.
> Sometimes this web server is restarting automatically, we don't have
> anything logged in the logs of deployed applications.
>
> Want to know the reason of this web server restart. The frequency is
> random, sometimes daily, sometimes twice a day, sometimes twice a week.
> Can someone help in understanding what is the problem and what else has to
> be checked.
>
> Java version - jdk1.8.0_333
> Tomcat version - apache-tomcat-8.5.79
>
> *web server restart logs*
> {root} #: journalctl -u abc-webserver
>
> Mar 01 15:34:45 rc5abcco systemd[1]: abc-webserver.service: main
> process exited, code=killed, status=6/ABRT
> Mar 01 15:34:45 rc5abcco systemd[1]: Unit abc-webserver.service entered
> failed state.
> Mar 01 15:34:45 rc5abcco systemd[1]: abc-webserver.service failed.
> Mar 01 15:35:00 rc5abcco systemd[1]: abc-webserver.service holdoff time
> over, scheduling restart.
> Mar 01 15:35:00 rc5abcco systemd[1]: Stopped The abc Webserver
> Service.
> Mar 01 15:35:00 rc5abcco systemd[1]: Starting The abc Webserver
> Service...
> Mar 01 15:35:12 rc5abcco abc_smf_start_stop.sh[2532]: webserver is
> running OK
> Mar 01 15:35:12 rc5abcco systemd[1]: Started The abc Webserver Service.
> Mar 01 22:33:36 rc5abcco systemd[1]: abc-webserver.service: main
> process exited, code=killed, status=6/ABRT
> Mar 01 22:33:36 rc5abcco systemd[1]: Unit abc-webserver.service entered
> failed state.
> Mar 01 22:33:36 rc5abcco systemd[1]: abc-webserver.service failed.
> Mar 01 22:33:51 rc5abcco systemd[1]: abc-webserver.service holdoff time
> over, scheduling restart.
> Mar 01 22:33:51 rc5abcco systemd[1]: Stopped The abc Webserver
> Service.
> Mar 01 22:33:51 rc5abcco systemd[1]: Starting The abc Webserver
> Service...
> Mar 01 22:34:02 rc5abcco abc_smf_start_stop.sh[44678]: webserver is
> running OK
> Mar 01 22:34:02 rc5abcco systemd[1]: Started The abc Webserver Service.
> .
> .
> .
> Jun 01 09:48:58 rc5abcco systemd[1]: Stopped The abc Webserver Service.
> Jun 01 09:49:10 rc5abcco systemd[1]: Started The abcWebserver Service.
>
> We have Tomcat web server as a service that has 4 web applications
> deployed on it. One of them is ALARMCFG, an alarm reporting application.
> Sometimes this web server is restarting automatically, we don't have
> anything logged in the logs of deployed applications.
>
> Want to know the reason of this web server restart. The frequency is
> random, sometimes daily, sometimes twice a day, sometimes twice a week.
> Can someone help in understanding what is the problem and what else has to
> be checked.
>
> Java version - jdk1.8.0_333
> Tomcat version - apache-tomcat-8.5.79
> web server restart logs
>
>
> {root} #: journalctl -u abc-webserver
>
> Mar 01 15:34:45 rc5abcco systemd[1]: abc-webserver.service: main
> process exited, code=killed, status=6/ABRT
> Mar 01 15:34:45 rc5abcco systemd[1]: Unit abc-webserver.service entered
> failed state.
> Mar 01 15:34:45 rc5abcco systemd[1]: abc-webserver.service failed.
> Mar 01 15:35:00 rc5abcco systemd[1]: abc-webserver.service holdoff time
> over, scheduling restart.
> Mar 01 15:35:00 rc5abcco systemd[1]: Stopped The abc Webserver
> Service.
> Mar 01 15:35:00 rc5abcco systemd[1]: Starting The abc Webserver
> Service...
> Mar 01 15:35:12 rc5abcco abc_smf_start_stop.sh[2532]: webserver is
> running OK
> Mar 01 15:35:12 rc5abcco systemd[1]: Started The abc Webserver Service.
> Mar 01 22:33:36 rc5abcco systemd[1]: abc-webserver.service: main
> process exited, code=killed, status=6/ABRT
> Mar 01 22:33:36 rc5abcco systemd[1]: Unit abc-webserver.service entered
> failed state.
> Mar 01 22:33:36 rc5abcco systemd[1]: abc-webserver.service failed.
> Mar 01 22:33:51 rc5abcco systemd[1]: abc-webserver.service holdoff time
> over, scheduling restart.
> Mar 01 22:33:51 rc5abcco systemd[1]: Stopped The abc Webserver
> Service.
> Mar 01 22:33:51 rc5abcco systemd[1]: Starting The abc Webserver
> Service...
> Mar 01 22:34:02 rc5abcco abc_smf_start_stop.sh[44678]: webserver is
> running OK
> Mar 01 22:34:02 rc5abcco systemd[1]: Started The abc Webserver Service.
> .
> .
> .
> Jun 01 09:48:58 rc5abcco systemd[1]: Stopped The abc Webserver Service.
> Jun 01 09:49:10 rc5abcco systemd[1]: Started The abcWebserver Service
>
> *Catalina.out*
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x00007fe638000368, pid=54728,
> tid=0x00007fe625309700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_333-b31) (build
> 1.8.0_333-b31)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.333-b31 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # C 0x00007fe638000368
> #
> # Failed to write core dump. Core dumps have been disabled. To enable
> core dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /tmp/hs_err_pid54728.log
> [thread 140626492360448 also had an error]
> [thread 140626582980352 also had an error][thread 140626573510400
> also had an error][thread 140626420045568 also had an error][thread
> 140626442094336 also had an error]
>
>
>
> [thread 140626437883648 also had an error]
> [thread 140626548127488 also had an error][thread 140626414782208
> also had an error][thread 140626567083776 also had an error]
>
>
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.java.com/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> Jun 01, 2023 9:49:00 AM
> org.apache.tomcat.util.digester.SetPropertiesRule begin
> WARNING: [SetPropertiesRule]{Server} Setting property 'command' to
> 'SHUTDOWN' did not find a matching property.
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.mbeans.AbcGlobalResourcesLifecycleListener
> INFO: AbcUserDatabaseMbean descriptor loaded
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: Server version name: Apache Tomcat
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: Server built: May 16 2022 15:36:23 UTC
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: Server version number: 0.0.0.0
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: OS Name: Linux
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: OS Version: 3.10.0-1160.80.1.el7.x86_64
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: Architecture: amd64
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: Java Home: /abc/sw/runtime/jdk1.8.0_333/jre
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: JVM Version: 1.8.0_333-b31
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: JVM Vendor: Oracle Corporation
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: CATALINA_BASE: /abc/sw/runtime/apache-tomcat-8.5.79
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: CATALINA_HOME: /abc/sw/runtime/apache-tomcat-8.5.79
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: Command line argument: -Xmx512M
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: Command line argument: -DHEAP_SIZE=512M
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: Command line argument: -Dpname=WebServer
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: Command line argument: -Djava.endorsed.dirs=
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: Command line argument: -Dcatalina.base=/abc/sw/runtime/tomcat
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: Command line argument: -
> Dcatalina.home=/abc/sw/runtime/tomcat
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: Command line argument:
> -Djava.io.tmpdir=/abc/sw/runtime/tomcat/temp
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: Command line argument: -DLOG_DIR=/abc/log/sw_log
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: Command line argument: -DCONF_DIR=/abc/sw/conf
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: Command line argument: -DRT_DIR=/abc/sw/runtime
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: Command line argument:
> -Djava.rmi.server.codebase=file:///abc/sw/platform/common-
> R29H01_EC01b1486/dclib/common.jar
> file:///abc/sw/platform/engine-R32H01_EC05b3632/dclib/engine.jar
> file:///abc/sw/platform/licensing-R26G01b491/dclib/licensing.jar
> file:///abc/sw/platform/scheduler-R25H04b525/dclib/scheduler.jar
> file:///abc/sw/platform/symboliclinkcreator-
> R19H09_EC02b369/dclib/symboliclinkcreator.jar
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: Command line argument:
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> Jun 01, 2023 9:49:00 AM
> org.apache.catalina.startup.VersionLoggerListener log
> INFO: Command line argument:
> -Dlog4j2.configurationFile=file:/abc/sw/runtime/apache-tomcat-
> 8.5.79/webapps/adminui/conf/adminui.properties
> Jun 01, 2023 9:49:00 AM org.apache.catalina.core.AprLifecycleListener
> lifecycleEvent
> Jun 01, 2023 9:49:00 AM org.apache.coyote.AbstractProtocol init
> INFO: Initializing ProtocolHandler ["https-jsse-nio-8443"]
> Jun 01, 2023 9:49:00 AM org.apache.catalina.startup.Catalina load
> INFO: Initialization processed in 678 ms
> Jun 01, 2023 9:49:00 AM
> com.abcdef.catalina.users.AbcUserDatabaseFactory getObjectInstance
> INFO: ABC UserDatabase successfully created.
> Jun 01, 2023 9:49:00 AM org.apache.catalina.core.StandardService
> startInternal
> INFO: Starting service [Catalina]
> Jun 01, 2023 9:49:00 AM org.apache.catalina.core.StandardEngine
> startInternal
> INFO: Starting Servlet engine: [Apache Tomcat]
> Jun 01, 2023 9:49:00 AM org.apache.catalina.startup.HostConfig
> deployDirectory
> INFO: Deploying web application directory [/abc/sw/runtime/apache-
> tomcat-8.5.79/webapps/ROOT]
> Jun 01, 2023 9:49:01 AM org.apache.jasper.servlet.TldScanner scanJars
> INFO: At least one JAR was scanned for TLDs yet contained no TLDs.
> Enable debug logging for this logger for a complete list of JARs that were
> scanned but no TLDs were found in them. Skipping unneeded JARs during
> scanning can improve startup time and JSP compilation time.
> Jun 01, 2023 9:49:01 AM org.apache.catalina.startup.HostConfig
> deployDirectory
> INFO: Deployment of web application directory [/abc/sw/runtime/apache-
> tomcat-8.5.79/webapps/ROOT] has finished in [616] ms
> Jun 01, 2023 9:49:01 AM org.apache.catalina.startup.HostConfig
> deployDirectory
> INFO: Deploying web application directory [/abc/sw/runtime/apache-
> tomcat-8.5.79/webapps/host-manager]
> Jun 01, 2023 9:49:01 AM org.apache.jasper.servlet.TldScanner scanJars
> INFO: At least one JAR was scanned for TLDs yet contained no TLDs.
> Enable debug logging for this logger for a complete list of JARs that were
> scanned but no TLDs were found in them. Skipping unneeded JARs during
> scanning can improve startup time and JSP compilation time.
> Jun 01, 2023 9:49:01 AM org.apache.catalina.startup.HostConfig
> deployDirectory
> INFO: Deployment of web application directory [/abc/sw/runtime/apache-
> tomcat-8.5.79/webapps/host-manager] has finished in [53] ms
> Jun 01, 2023 9:49:01 AM org.apache.catalina.startup.HostConfig
> deployDirectory
> INFO: Deploying web application directory [/abc/sw/runtime/apache-
> tomcat-8.5.79/webapps/manager]
> Jun 01, 2023 9:49:01 AM org.apache.jasper.servlet.TldScanner scanJars
> INFO: At least one JAR was scanned for TLDs yet contained no TLDs.
> Enable debug logging for this logger for a complete list of JARs that were
> scanned but no TLDs were found in them. Skipping unneeded JARs during
> scanning can improve startup time and JSP compilation time.
> Jun 01, 2023 9:49:01 AM org.apache.catalina.startup.HostConfig
> deployDirectory
> INFO: Deployment of web application directory [/abc/sw/runtime/apache-
> tomcat-8.5.79/webapps/manager] has finished in [56] ms
> Jun 01, 2023 9:49:01 AM org.apache.catalina.startup.HostConfig
> deployDirectory
> INFO: Deploying web application directory [/abc/sw/runtime/apache-
> tomcat-8.5.79/webapps/adminui]
> Jun 01, 2023 9:49:02 AM org.apache.catalina.startup.HostConfig
> deployDirectory
> INFO: Deployment of web application directory [/abc/sw/runtime/apache-
> tomcat-8.5.79/webapps/adminui] has finished in [1,244] ms
> Jun 01, 2023 9:49:02 AM org.apache.catalina.startup.HostConfig
> deployDirectory
> INFO: Deploying web application directory [/abc/sw/runtime/apache-
> tomcat-8.5.79/webapps/alarmcfg]
> Jun 01, 2023 9:49:16 AM org.apache.catalina.startup.HostConfig
> deployDirectory
> INFO: Deployment of web application directory [/abc/sw/runtime/apache-
> tomcat-8.5.79/webapps/alarmcfg] has finished in [13,051] ms
> Jun 01, 2023 9:49:16 AM org.apache.catalina.startup.HostConfig
> deployDirectory
> INFO: Deploying web application directory [/abc/sw/runtime/apache-
> tomcat-8.5.79/webapps/universeupdateinstaller]
> Jun 01, 2023 9:49:16 AM org.apache.jasper.servlet.TldScanner scanJars
> INFO: At least one JAR was scanned for TLDs yet contained no TLDs.
> Enable debug logging for this logger for a complete list of JARs that were
> scanned but no TLDs were found in them. Skipping unneeded JARs during
> scanning can improve startup time and JSP compilation time.
> Jun 01, 2023 9:49:16 AM org.apache.catalina.startup.HostConfig
> deployDirectory
> INFO: Deployment of web application directory [/abc/sw/runtime/apache-
> tomcat-8.5.79/webapps/universeupdateinstaller] has finished in [274] ms
> Jun 01, 2023 9:49:16 AM org.apache.coyote.AbstractProtocol start
> INFO: Starting ProtocolHandler ["https-jsse-nio-8443"]
> Jun 01, 2023 9:49:16 AM org.apache.catalina.startup.Catalina start
> INFO: Server startup in 15359 ms
>
> *hs_err_pid54728.log*
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=**0x00007fe638000368**, pid=54728,
> tid=0x00007fe625309700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_333-b31) (build
> 1.8.0_333-b31)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.333-b31 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # C **0x00007fe638000368**
> #
> # Failed to write core dump. Core dumps have been disabled. To enable
> core dumping, try "ulimit -c unlimited" before starting Java again
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.java.com/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
>
> --------------- T H R E A D ---------------
>
> Current thread (0x00007fe641393800): JavaThread "https-jsse-nio-8443-
> exec-15121" daemon [_thread_in_native, id=63271,
> stack(0x00007fe625209000,0x00007fe62530a000)]
>
> siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr:
> **0x00007fe638000368**
>
> Registers:
> RAX=0x00007fe638000318, RBX=0x00000000a010000a,
> RCX=0x0000000000000010,
> RDX=0x00007fe625307bc8
> RSP=0x00007fe625307ba8, RBP=0x00007fe625307c18,
> RSI=0x00007fe638f80fc0,
> RDI=0x00007fe638daf530
> R8 =0x0000000000000000, R9 =0x0000000004000001,
> R10=0x0000000000000001,
> R11=0x0000000000000283
> R12=0x00007fe641393a00, R13=0x00000000fc9669b0,
> R14=0x0000000000000000,
> R15=0x00007fe641393800
> RIP=**0x00007fe638000368**, EFLAGS=0x0000000000010206,
> CSGSFS=0x0000000000000033, ERR=0x0000000000000015
> TRAPNO=0x000000000000000e
>
> Top of Stack: (sp=0x00007fe625307ba8)
> 0x00007fe625307ba8: 00007fe62bf81fec 00007fe625307c18
> 0x00007fe625307bb8: 00007fe638f80fc0 00007fe638f80fc0
> 0x00007fe625307bc8: 0000000000000042 0000000000000042
> 0x00007fe625307bd8: 00007fe638daf530 00007fe62f31e950
> 0x00007fe625307be8: 00007fe625307c50 0000000000000000
>
> Regards,
> Thomson
It seems the JVM is crashing and system is then restarting the service.
It doesn't look like a Tomcat issue but rather an issue with the JVM.
Maybe the core dump gives more insight. The error explains what to do (ulimit
-c unlimited).
The key might be to inspect the hs_err_pidxxx.log files with the generated core
dump.
Greetings,
Thomas
