Hi Jaejong,
Are you able to start clouddtack service without any errors? If not check the
service using port 9090 and disable/stop it.
systemctl list-sockets
-Jithin
From: jaejong <cb...@naver.com>
Date: Wednesday, 20 September 2023 at 10:06 AM
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: Re: Detected that another management node with the same IP is already
running, please check your cluster configuration
hellow Jithin
I reboot host with cloudstack-management enabled
port status are :
$ sudo netstat -nap | grep 8080
tcp6 0 0 :::8080 :::* LISTEN
1147/java
tcp6 0 0 :::9090 :::* LISTEN
1/systemd
tcp6 0 0 10.0.33.1:9090 10.0.33.1:47813 TIME_WAIT
-
tcp6 0 0 10.0.33.1:9090 10.0.33.1:42097 TIME_WAIT
-
tcp6 0 0 10.0.33.1:9090 10.0.33.1:44439 TIME_WAIT
-
tcp6 0 0 10.0.33.1:9090 10.0.33.1:52903 TIME_WAIT
-
tcp6 0 0 10.0.33.1:9090 10.0.33.1:33893 TIME_WAIT
-
tcp6 0 0 10.0.33.1:9090 10.0.33.1:45235 TIME_WAIT
-
tcp6 0 0 10.0.33.1:9090 10.0.33.1:60319 TIME_WAIT
-
tcp6 0 0 10.0.33.1:9090 10.0.33.1:49279 TIME_WAIT
-
tcp6 0 0 :::8250 :::* LISTEN
1147/java
But after rebooting with cloudstack-management disabled
only 9090 is used.
$ sudo netstat -nap | grep 9090
tcp6 0 0 :::9090 :::* LISTEN 1/systemd
thanks a lot.
-----Original Message-----
From: "Jithin Raju"<jithin.r...@shapeblue.com>
To: "users@cloudstack.apache.org"<users@cloudstack.apache.org>;
Cc:
Sent: 2023-09-20 (수) 12:46:36 (GMT+09:00)
Subject: Re: Detected that another management node with the same IP is already
running, please check your cluster configuration
Hi Jaejong,
Could you check whether any of the port numbers 8250,9090,8080 TCP are in use
already? Or reboot the OS?
-Jithin
From: jaejong <cb...@naver.com>
Date: Tuesday, 19 September 2023 at 4:20 PM
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: Detected that another management node with the same IP is already
running, please check your cluster configuration
rocky linux 9.2
mysql 8.0.32
A single Management Server node with mysql on the same node
1. After rebooting management host I get follow error messages
sudo systemctl status cloudstack-management
Loaded: loaded (/usr/lib/systemd/system/cloudstack-management.service;
enabled; preset: disabled)
Active: active (running) since Tue 2023-09-19 17:35:39 KST; 2min 3s ago
Main PID: 1148 (java)
Tasks: 49 (limit: 408699)
Memory: 1.1G
CPU: 40.653s
CGroup: /system.slice/cloudstack-management.service
└─1148 /usr/bin/java
-Djava.security.properties=/etc/cloudstack/management/java.security.ciphers
-Djava.awt.headless=true -Dcom.sun.management.>
java[1148]: at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
java[1148]: at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
java[1148]: at org.eclipse.jetty.server.Server.start(Server.java:423)
java[1148]: at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
java[1148]: at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
java[1148]: at org.eclipse.jetty.server.Server.doStart(Server.java:387)
java[1148]: at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
java[1148]: at
org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:192)
java[1148]: at
org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:107)
java[1148]: INFO [o.a.c.s.NfsMountManager] (main:null) (logid:) Clean up
mounted NFS mount points used in current session
server.log
2023-09-19 17:35:54,352 ERROR [c.c.c.ClusterManagerImpl] (main:null) (logid:)
Detected that another management node with the same IP 10.0.33.1 is already
running, please check your cluster configuration
2023-09-19 17:35:54,353 ERROR [o.a.c.s.l.CloudStackExtendedLifeCycle]
(main:null) (logid:) Failed to configure ClusterManagerImpl
javax.naming.ConfigurationException: Detected that another management node with
the same IP is already running, please check your cluster configuration
at
com.cloud.cluster.ClusterManagerImpl.checkConflicts(ClusterManagerImpl.java:1245)
at
com.cloud.cluster.ClusterManagerImpl.configure(ClusterManagerImpl.java:1115)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle$3.with(CloudStackExtendedLifeCycle.java:114)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.with(CloudStackExtendedLifeCycle.java:153)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.configure(CloudStackExtendedLifeCycle.java:110)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:55)
at
org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178)
at
org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54)
at
org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at
org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155)
at
org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123)
at
org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:937)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:144)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:121)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:244)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:249)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:249)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:232)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:116)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:78)
at
org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:70)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:57)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:61)
at
org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52)
at
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
at
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at
org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
at
org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:765)
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
at
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:423)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:387)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:192)
at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:107)
2. After restart cloudstack-management
2-1 as soon as restart
● cloudstack-management.service - CloudStack Management Server
Loaded: loaded (/usr/lib/systemd/system/cloudstack-management.service;
enabled; preset: disabled)
Active: active (running) since Tue 2023-09-19 17:47:21 KST; 6s ago
Main PID: 1893 (java)
Tasks: 43 (limit: 408699)
Memory: 895.5M
CPU: 19.833s
CGroup: /system.slice/cloudstack-management.service
└─1893 /usr/bin/java
-Djava.security.properties=/etc/cloudstack/management/java.security.ciphers
-Djava.awt.headless=true -Dcom.sun.management.>
Sep 19 17:47:25 mgmt1.acs.unitm.kr java[1893]: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module
context [core] from URL >
Sep 19 17:47:25 mgmt1.acs.unitm.kr java[1893]: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module
context [core] from URL >
Sep 19 17:47:25 mgmt1.acs.unitm.kr java[1893]: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module
context [core] from URL >
Sep 19 17:47:25 mgmt1.acs.unitm.kr java[1893]: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module
context [core] from URL >
Sep 19 17:47:25 mgmt1.acs.unitm.kr java[1893]: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module
context [core] from URL >
Sep 19 17:47:25 mgmt1.acs.unitm.kr java[1893]: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module
context [core] from URL >
Sep 19 17:47:25 mgmt1.acs.unitm.kr java[1893]: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module
context [core] from URL >
Sep 19 17:47:25 mgmt1.acs.unitm.kr java[1893]: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module
context [core] from URL >
Sep 19 17:47:25 mgmt1.acs.unitm.kr java[1893]: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module
context [core] from URL >
Sep 19 17:47:28 mgmt1.acs.unitm.kr java[1893]: INFO [c.c.s.GsonHelper]
(main:null) (logid:) Default Builder inited.
2-2 after some seconds
● cloudstack-management.service - CloudStack Management Server
Loaded: loaded (/usr/lib/systemd/system/cloudstack-management.service;
enabled; preset: disabled)
Active: active (running) since Tue 2023-09-19 17:47:21 KST; 1min 27s ago
Main PID: 1893 (java)
Tasks: 183 (limit: 408699)
Memory: 1.0G
CPU: 56.129s
CGroup: /system.slice/cloudstack-management.service
└─1893 /usr/bin/java
-Djava.security.properties=/etc/cloudstack/management/java.security.ciphers
-Djava.awt.headless=true -Dcom.sun.management.>
Sep 19 17:47:39 mgmt1.acs.unitm.kr java[1893]: INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-cfbc07ba) (logid:ce3e9ffc) We are good, no orphan m>
Sep 19 17:47:39 mgmt1.acs.unitm.kr java[1893]: INFO [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-cfbc07ba) (logid:ce3e9ffc) No inactive management s>
Sep 19 17:47:41 mgmt1.acs.unitm.kr java[1893]: WARN [c.c.a.AlertManagerImpl]
(Cluster-Notification-1:ctx-d247bd11) (logid:f2700a2b) alertType=[14] dataCent>
Sep 19 17:47:41 mgmt1.acs.unitm.kr java[1893]: WARN [c.c.a.AlertManagerImpl]
(Cluster-Notification-1:ctx-d247bd11) (logid:f2700a2b) No recipients set in gl>
Sep 19 17:47:41 mgmt1.acs.unitm.kr java[1893]: WARN [c.c.c.ClusterManagerImpl]
(Cluster-Notification-1:ctx-d247bd11) (logid:f2700a2b) Notifying management >
Sep 19 17:48:39 mgmt1.acs.unitm.kr java[1893]: INFO [c.c.s.StatsCollector]
(StatsCollector-1:ctx-fb9c4c33) (logid:891ff247) system memory from /proc:
67003>
Sep 19 17:48:39 mgmt1.acs.unitm.kr java[1893]: INFO [c.c.s.StatsCollector]
(StatsCollector-1:ctx-fb9c4c33) (logid:891ff247) free memory from /proc:
6451747>
Sep 19 17:48:39 mgmt1.acs.unitm.kr java[1893]: INFO [c.c.s.StatsCollector]
(StatsCollector-1:ctx-fb9c4c33) (logid:891ff247) used memory from /proc: 1119316
Sep 19 17:48:39 mgmt1.acs.unitm.kr java[1893]: ERROR
[c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:ctx-25684151)
(logid:c61e1671) IOException from : h>
Sep 19 17:48:39 mgmt1.acs.unitm.kr java[1893]: ERROR
[c.c.c.ClusterServiceServletImpl] (Cluster-Worker-1:ctx-25684151)
(logid:c61e1671) IOException from : h>
server logs
2023-09-19 17:47:34,671 ERROR [c.c.u.PropertiesUtil] (main:null) (logid:)
Unable to find properties file: commands.properties
2023-09-19 17:47:38,379 ERROR [c.c.c.ClusterServiceServletContainer]
(main:null) (logid:) error initializing cluster service servlet container
java.net.BindException: Address already in use (Bind failed)
at java.base/java.net.PlainSocketImpl.socketBind(Native Method)
at
java.base/java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:452)
at java.base/java.net.ServerSocket.bind(ServerSocket.java:395)
at java.base/java.net.ServerSocket.<init>(ServerSocket.java:257)
at java.base/java.net.ServerSocket.<init>(ServerSocket.java:149)
at
com.cloud.cluster.ClusterServiceServletContainer$ListenerThread.<init>(ClusterServiceServletContainer.java:82)
at
com.cloud.cluster.ClusterServiceServletContainer.start(ClusterServiceServletContainer.java:60)
at
com.cloud.cluster.ClusterServiceServletAdapter.start(ClusterServiceServletAdapter.java:111)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle$1.with(CloudStackExtendedLifeCycle.java:74)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.with(CloudStackExtendedLifeCycle.java:153)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.startBeans(CloudStackExtendedLifeCycle.java:71)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycleStart.run(CloudStackExtendedLifeCycleStart.java:46)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$1.with(DefaultModuleDefinitionSet.java:104)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:244)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:249)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:249)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:232)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.startContexts(DefaultModuleDefinitionSet.java:96)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:79)
at
org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:70)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:57)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:61)
at
org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52)
at
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
at
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at
org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
at
org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:765)
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
at
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:423)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:387)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:192)
at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:107)
2023-09-19 17:47:39,549 ERROR [c.c.c.ClusterServiceServletImpl]
(Cluster-Worker-5:ctx-9f9b2994) (logid:66f4952c) IOException from :
http://hostip:9090/clusterservice, method : name=method, value=5
2023-09-19 17:47:39,555 ERROR [c.c.c.ClusterServiceServletImpl]
(Cluster-Worker-5:ctx-9f9b2994) (logid:66f4952c) IOException from :
http://hostip:9090/clusterservice, method : name=method, value=5
....
I saw issue #6814 that I think it is same problem, so I check database;
after reboot first db status
select * from mshost;
state up, port 9090
select * from mshost_peer;
state up
after restart cloudstack-management
always mshost is down state but mshost_peer up state even though
cloudstack-management stoped.
