Hi Team,
We have Tomcat web server as a service that has 4 web applications deployed
on it. One of them is ALARMCFG, an alarm reporting application. Sometimes
this web server is restarting automatically, we don't have anything logged
in the logs of deployed applications.
Want to know the reason of this web server restart. The frequency is
random, sometimes daily, sometimes twice a day, sometimes twice a week. Can
someone help in understanding what is the problem and what else has to be
checked.
Java version - jdk1.8.0_333
Tomcat version - apache-tomcat-8.5.79
*web server restart logs*
{root} #: journalctl -u abc-webserver
Mar 01 15:34:45 rc5abcco systemd[1]: abc-webserver.service: main
process exited, code=killed, status=6/ABRT
Mar 01 15:34:45 rc5abcco systemd[1]: Unit abc-webserver.service entered
failed state.
Mar 01 15:34:45 rc5abcco systemd[1]: abc-webserver.service failed.
Mar 01 15:35:00 rc5abcco systemd[1]: abc-webserver.service holdoff time
over, scheduling restart.
Mar 01 15:35:00 rc5abcco systemd[1]: Stopped The abc Webserver Service.
Mar 01 15:35:00 rc5abcco systemd[1]: Starting The abc Webserver
Service...
Mar 01 15:35:12 rc5abcco abc_smf_start_stop.sh[2532]: webserver is
running OK
Mar 01 15:35:12 rc5abcco systemd[1]: Started The abc Webserver Service.
Mar 01 22:33:36 rc5abcco systemd[1]: abc-webserver.service: main
process exited, code=killed, status=6/ABRT
Mar 01 22:33:36 rc5abcco systemd[1]: Unit abc-webserver.service entered
failed state.
Mar 01 22:33:36 rc5abcco systemd[1]: abc-webserver.service failed.
Mar 01 22:33:51 rc5abcco systemd[1]: abc-webserver.service holdoff time
over, scheduling restart.
Mar 01 22:33:51 rc5abcco systemd[1]: Stopped The abc Webserver Service.
Mar 01 22:33:51 rc5abcco systemd[1]: Starting The abc Webserver
Service...
Mar 01 22:34:02 rc5abcco abc_smf_start_stop.sh[44678]: webserver is
running OK
Mar 01 22:34:02 rc5abcco systemd[1]: Started The abc Webserver Service.
.
.
.
Jun 01 09:48:58 rc5abcco systemd[1]: Stopped The abc Webserver Service.
Jun 01 09:49:10 rc5abcco systemd[1]: Started The abcWebserver Service.
We have Tomcat web server as a service that has 4 web applications deployed
on it. One of them is ALARMCFG, an alarm reporting application. Sometimes
this web server is restarting automatically, we don't have anything logged
in the logs of deployed applications.
Want to know the reason of this web server restart. The frequency is
random, sometimes daily, sometimes twice a day, sometimes twice a week. Can
someone help in understanding what is the problem and what else has to be
checked.
Java version - jdk1.8.0_333
Tomcat version - apache-tomcat-8.5.79
web server restart logs
{root} #: journalctl -u abc-webserver
Mar 01 15:34:45 rc5abcco systemd[1]: abc-webserver.service: main
process exited, code=killed, status=6/ABRT
Mar 01 15:34:45 rc5abcco systemd[1]: Unit abc-webserver.service entered
failed state.
Mar 01 15:34:45 rc5abcco systemd[1]: abc-webserver.service failed.
Mar 01 15:35:00 rc5abcco systemd[1]: abc-webserver.service holdoff time
over, scheduling restart.
Mar 01 15:35:00 rc5abcco systemd[1]: Stopped The abc Webserver Service.
Mar 01 15:35:00 rc5abcco systemd[1]: Starting The abc Webserver
Service...
Mar 01 15:35:12 rc5abcco abc_smf_start_stop.sh[2532]: webserver is
running OK
Mar 01 15:35:12 rc5abcco systemd[1]: Started The abc Webserver Service.
Mar 01 22:33:36 rc5abcco systemd[1]: abc-webserver.service: main
process exited, code=killed, status=6/ABRT
Mar 01 22:33:36 rc5abcco systemd[1]: Unit abc-webserver.service entered
failed state.
Mar 01 22:33:36 rc5abcco systemd[1]: abc-webserver.service failed.
Mar 01 22:33:51 rc5abcco systemd[1]: abc-webserver.service holdoff time
over, scheduling restart.
Mar 01 22:33:51 rc5abcco systemd[1]: Stopped The abc Webserver Service.
Mar 01 22:33:51 rc5abcco systemd[1]: Starting The abc Webserver
Service...
Mar 01 22:34:02 rc5abcco abc_smf_start_stop.sh[44678]: webserver is
running OK
Mar 01 22:34:02 rc5abcco systemd[1]: Started The abc Webserver Service.
.
.
.
Jun 01 09:48:58 rc5abcco systemd[1]: Stopped The abc Webserver Service.
Jun 01 09:49:10 rc5abcco systemd[1]: Started The abcWebserver Service
*Catalina.out*
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fe638000368, pid=54728,
tid=0x00007fe625309700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_333-b31) (build
1.8.0_333-b31)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.333-b31 mixed mode
linux-amd64 compressed oops)
# Problematic frame:
# C 0x00007fe638000368
#
# Failed to write core dump. Core dumps have been disabled. To enable
core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid54728.log
[thread 140626492360448 also had an error]
[thread 140626582980352 also had an error][thread 140626573510400 also
had an error][thread 140626420045568 also had an error][thread
140626442094336 also had an error]
[thread 140626437883648 also had an error]
[thread 140626548127488 also had an error][thread 140626414782208 also
had an error][thread 140626567083776 also had an error]
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Jun 01, 2023 9:49:00 AM
org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server} Setting property 'command' to
'SHUTDOWN' did not find a matching property.
Jun 01, 2023 9:49:00 AM
org.apache.catalina.mbeans.AbcGlobalResourcesLifecycleListener
INFO: AbcUserDatabaseMbean descriptor loaded
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: Server version name: Apache Tomcat
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: Server built: May 16 2022 15:36:23 UTC
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: Server version number: 0.0.0.0
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: OS Name: Linux
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: OS Version: 3.10.0-1160.80.1.el7.x86_64
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: Architecture: amd64
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: Java Home: /abc/sw/runtime/jdk1.8.0_333/jre
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: JVM Version: 1.8.0_333-b31
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: JVM Vendor: Oracle Corporation
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: CATALINA_BASE: /abc/sw/runtime/apache-tomcat-8.5.79
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: CATALINA_HOME: /abc/sw/runtime/apache-tomcat-8.5.79
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Xmx512M
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -DHEAP_SIZE=512M
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Dpname=WebServer
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Djava.endorsed.dirs=
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Dcatalina.base=/abc/sw/runtime/tomcat
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Dcatalina.home=/abc/sw/runtime/tomcat
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument:
-Djava.io.tmpdir=/abc/sw/runtime/tomcat/temp
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -DLOG_DIR=/abc/log/sw_log
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -DCONF_DIR=/abc/sw/conf
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -DRT_DIR=/abc/sw/runtime
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument:
-Djava.rmi.server.codebase=file:///abc/sw/platform/common-R29H01_EC01b1486/dclib/common.jar
file:///abc/sw/platform/engine-R32H01_EC05b3632/dclib/engine.jar
file:///abc/sw/platform/licensing-R26G01b491/dclib/licensing.jar
file:///abc/sw/platform/scheduler-R25H04b525/dclib/scheduler.jar
file:///abc/sw/platform/symboliclinkcreator-R19H09_EC02b369/dclib/symboliclinkcreator.jar
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument:
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
Jun 01, 2023 9:49:00 AM
org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument:
-Dlog4j2.configurationFile=file:/abc/sw/runtime/apache-tomcat-8.5.79/webapps/adminui/conf/adminui.properties
Jun 01, 2023 9:49:00 AM org.apache.catalina.core.AprLifecycleListener
lifecycleEvent
Jun 01, 2023 9:49:00 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["https-jsse-nio-8443"]
Jun 01, 2023 9:49:00 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 678 ms
Jun 01, 2023 9:49:00 AM
com.abcdef.catalina.users.AbcUserDatabaseFactory getObjectInstance
INFO: ABC UserDatabase successfully created.
Jun 01, 2023 9:49:00 AM org.apache.catalina.core.StandardService
startInternal
INFO: Starting service [Catalina]
Jun 01, 2023 9:49:00 AM org.apache.catalina.core.StandardEngine
startInternal
INFO: Starting Servlet engine: [Apache Tomcat]
Jun 01, 2023 9:49:00 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory
[/abc/sw/runtime/apache-tomcat-8.5.79/webapps/ROOT]
Jun 01, 2023 9:49:01 AM org.apache.jasper.servlet.TldScanner scanJars
INFO: At least one JAR was scanned for TLDs yet contained no TLDs.
Enable debug logging for this logger for a complete list of JARs that were
scanned but no TLDs were found in them. Skipping unneeded JARs during
scanning can improve startup time and JSP compilation time.
Jun 01, 2023 9:49:01 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deployment of web application directory
[/abc/sw/runtime/apache-tomcat-8.5.79/webapps/ROOT] has finished in [616] ms
Jun 01, 2023 9:49:01 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory
[/abc/sw/runtime/apache-tomcat-8.5.79/webapps/host-manager]
Jun 01, 2023 9:49:01 AM org.apache.jasper.servlet.TldScanner scanJars
INFO: At least one JAR was scanned for TLDs yet contained no TLDs.
Enable debug logging for this logger for a complete list of JARs that were
scanned but no TLDs were found in them. Skipping unneeded JARs during
scanning can improve startup time and JSP compilation time.
Jun 01, 2023 9:49:01 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deployment of web application directory
[/abc/sw/runtime/apache-tomcat-8.5.79/webapps/host-manager] has finished in
[53] ms
Jun 01, 2023 9:49:01 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory
[/abc/sw/runtime/apache-tomcat-8.5.79/webapps/manager]
Jun 01, 2023 9:49:01 AM org.apache.jasper.servlet.TldScanner scanJars
INFO: At least one JAR was scanned for TLDs yet contained no TLDs.
Enable debug logging for this logger for a complete list of JARs that were
scanned but no TLDs were found in them. Skipping unneeded JARs during
scanning can improve startup time and JSP compilation time.
Jun 01, 2023 9:49:01 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deployment of web application directory
[/abc/sw/runtime/apache-tomcat-8.5.79/webapps/manager] has finished in [56]
ms
Jun 01, 2023 9:49:01 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory
[/abc/sw/runtime/apache-tomcat-8.5.79/webapps/adminui]
Jun 01, 2023 9:49:02 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deployment of web application directory
[/abc/sw/runtime/apache-tomcat-8.5.79/webapps/adminui] has finished in
[1,244] ms
Jun 01, 2023 9:49:02 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory
[/abc/sw/runtime/apache-tomcat-8.5.79/webapps/alarmcfg]
Jun 01, 2023 9:49:16 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deployment of web application directory
[/abc/sw/runtime/apache-tomcat-8.5.79/webapps/alarmcfg] has finished in
[13,051] ms
Jun 01, 2023 9:49:16 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory
[/abc/sw/runtime/apache-tomcat-8.5.79/webapps/universeupdateinstaller]
Jun 01, 2023 9:49:16 AM org.apache.jasper.servlet.TldScanner scanJars
INFO: At least one JAR was scanned for TLDs yet contained no TLDs.
Enable debug logging for this logger for a complete list of JARs that were
scanned but no TLDs were found in them. Skipping unneeded JARs during
scanning can improve startup time and JSP compilation time.
Jun 01, 2023 9:49:16 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deployment of web application directory
[/abc/sw/runtime/apache-tomcat-8.5.79/webapps/universeupdateinstaller] has
finished in [274] ms
Jun 01, 2023 9:49:16 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["https-jsse-nio-8443"]
Jun 01, 2023 9:49:16 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 15359 ms
*hs_err_pid54728.log*
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=**0x00007fe638000368**, pid=54728,
tid=0x00007fe625309700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_333-b31) (build
1.8.0_333-b31)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.333-b31 mixed mode
linux-amd64 compressed oops)
# Problematic frame:
# C **0x00007fe638000368**
#
# Failed to write core dump. Core dumps have been disabled. To enable
core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
--------------- T H R E A D ---------------
Current thread (0x00007fe641393800): JavaThread
"https-jsse-nio-8443-exec-15121" daemon [_thread_in_native, id=63271,
stack(0x00007fe625209000,0x00007fe62530a000)]
siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr:
**0x00007fe638000368**
Registers:
RAX=0x00007fe638000318, RBX=0x00000000a010000a, RCX=0x0000000000000010,
RDX=0x00007fe625307bc8
RSP=0x00007fe625307ba8, RBP=0x00007fe625307c18, RSI=0x00007fe638f80fc0,
RDI=0x00007fe638daf530
R8 =0x0000000000000000, R9 =0x0000000004000001, R10=0x0000000000000001,
R11=0x0000000000000283
R12=0x00007fe641393a00, R13=0x00000000fc9669b0, R14=0x0000000000000000,
R15=0x00007fe641393800
RIP=**0x00007fe638000368**, EFLAGS=0x0000000000010206,
CSGSFS=0x0000000000000033, ERR=0x0000000000000015
TRAPNO=0x000000000000000e
Top of Stack: (sp=0x00007fe625307ba8)
0x00007fe625307ba8: 00007fe62bf81fec 00007fe625307c18
0x00007fe625307bb8: 00007fe638f80fc0 00007fe638f80fc0
0x00007fe625307bc8: 0000000000000042 0000000000000042
0x00007fe625307bd8: 00007fe638daf530 00007fe62f31e950
0x00007fe625307be8: 00007fe625307c50 0000000000000000
Regards,
Thomson
