Hello, I've recently installed a second management server ACS 4.16.1 following the installation instructions in section Additional Management Servers from the official documentation ( [ http://docs.cloudstack.apache.org/en/4.16.1.0/installguide/management-server/index.html | http://docs.cloudstack.apache.org/en/4.16.1.0/installguide/management-server/index.html ] ). I've installed the Ubuntu package on the second server of the same version as the primary management server. Configured the database with cloudstack-setup-databases command followed by running cloudstack-setup-management as per the documentation. There were no errors in the process and the cloudstack-management.service seems to have started just fine. The second ACS management service connected to the same database as the primary one and the login web GUI loaded just fine. The management server logs seems to show no apparent errors in the startup. The only exceptions I was getting in the logs were from the host agents showing status Disconnected.
So, I have tried to login (using domain and ROOT login accounts) to the web gui of the second management server and the page just hangs after I enter the credentials and press the Login button. I've tried several different browsers at no avail. Supplying the incorrect login credentials produce the error though. The management server logs do not show any errors during the login process. In fact, it seems that all commands produce " is allowed to perform API calls: 0.0.0.0/0,::/0 " message in the logs. There are no exceptions that I can see either: -------------- 2022-07-18 01:17:33,743 DEBUG [c.c.a.ApiServlet] (qtp681094281-285:ctx-0cf08734) (logid:94b277ba) ===START=== 192.168.169.251 -- POST 2022-07-18 01:17:33,750 DEBUG [c.c.u.AccountManagerImpl] (qtp681094281-285:ctx-0cf08734) (logid:94b277ba) Attempting to log in user: andrei in domain 1 2022-07-18 01:17:33,752 DEBUG [o.a.c.s.a.PBKDF2UserAuthenticator] (qtp681094281-285:ctx-0cf08734) (logid:94b277ba) Retrieving user: andrei 2022-07-18 01:17:33,969 DEBUG [c.c.u.AccountManagerImpl] (qtp681094281-285:ctx-0cf08734) (logid:94b277ba) CIDRs from which account 'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2, "name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is allowed to perform API calls: 0.0.0.0/0,::/0 2022-07-18 01:17:33,969 DEBUG [c.c.u.AccountManagerImpl] (qtp681094281-285:ctx-0cf08734) (logid:94b277ba) User: andrei in domain 1 has successfully logged in 2022-07-18 01:17:34,011 INFO [c.c.a.ApiServer] (qtp681094281-285:ctx-0cf08734) (logid:94b277ba) Current user logged in under Etc/UTC timezone 2022-07-18 01:17:34,011 INFO [c.c.a.ApiServer] (qtp681094281-285:ctx-0cf08734) (logid:94b277ba) Timezone offset from UTC is: 0.0 2022-07-18 01:17:34,015 DEBUG [c.c.a.ApiServlet] (qtp681094281-285:ctx-0cf08734) (logid:94b277ba) ===END=== 192.168.169.251 -- POST 2022-07-18 01:17:34,123 DEBUG [c.c.a.ApiServlet] (qtp681094281-280:ctx-fafe166c) (logid:41d7b4d5) ===START=== 192.168.169.251 -- GET listall=true&command=listZones&response=json 2022-07-18 01:17:34,133 DEBUG [c.c.a.ApiServer] (qtp681094281-280:ctx-fafe166c ctx-2269cc31) (logid:41d7b4d5) CIDRs from which account 'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2, "name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is allowed to perform API calls: 0.0.0.0/0,::/0 2022-07-18 01:17:34,133 DEBUG [c.c.a.ApiServlet] (qtp681094281-28:ctx-0906d03f) (logid:56b10f23) ===START=== 192.168.169.251 -- GET command=listApis&response=json 2022-07-18 01:17:34,137 DEBUG [c.c.a.ApiServlet] (qtp681094281-280:ctx-fafe166c ctx-2269cc31) (logid:41d7b4d5) ===END=== 192.168.169.251 -- GET listall=true&command=listZones&response=json 2022-07-18 01:17:34,144 DEBUG [c.c.a.ApiServer] (qtp681094281-28:ctx-0906d03f ctx-5a2a7dde) (logid:56b10f23) CIDRs from which account 'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2, "name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is allowed to perform API calls: 0.0.0.0/0,::/0 2022-07-18 01:17:34,153 DEBUG [c.c.a.ApiServlet] (qtp681094281-318:ctx-fc79b118) (logid:8a349f6d) ===START=== 192.168.169.251 -- GET command=cloudianIsEnabled&response=json 2022-07-18 01:17:34,163 DEBUG [c.c.a.ApiServer] (qtp681094281-318:ctx-fc79b118 ctx-40fd8f3a) (logid:8a349f6d) CIDRs from which account 'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2, "name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is allowed to perform API calls: 0.0.0.0/0,::/0 2022-07-18 01:17:34,168 DEBUG [c.c.a.ApiServlet] (qtp681094281-318:ctx-fc79b118 ctx-40fd8f3a) (logid:8a349f6d) ===END=== 192.168.169.251 -- GET command=cloudianIsEnabled&response=json 2022-07-18 01:17:34,176 DEBUG [c.c.a.ApiServlet] (qtp681094281-34:ctx-20a51695) (logid:2436a576) ===START=== 192.168.12022-07-18 01:17:34,123 DEBUG [c.c.a.ApiServlet] (qtp681094281-280:ctx-fafe166c) (logid:41d7b4d5) ===START=== 192.168.169.251 -- GET listall=true&command=listZones&response=json 2022-07-18 01:17:34,133 DEBUG [c.c.a.ApiServer] (qtp681094281-280:ctx-fafe166c ctx-2269cc31) (logid:41d7b4d5) CIDRs from which account 'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2, "name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is allowed to perform API calls: 0.0.0.0/0,::/0 2022-07-18 01:17:34,133 DEBUG [c.c.a.ApiServlet] (qtp681094281-28:ctx-0906d03f) (logid:56b10f23) ===START=== 192.168.169.251 -- GET command=listApis&response=json 2022-07-18 01:17:34,137 DEBUG [c.c.a.ApiServlet] (qtp681094281-280:ctx-fafe166c ctx-2269cc31) (logid:41d7b4d5) ===END=== 192.168.169.251 -- GET listall=true&command=listZones&response=json 2022-07-18 01:17:34,144 DEBUG [c.c.a.ApiServer] (qtp681094281-28:ctx-0906d03f ctx-5a2a7dde) (logid:56b10f23) CIDRs from which account 'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2, "name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is allowed to perform API calls: 0.0.0.0/0,::/0 2022-07-18 01:17:34,153 DEBUG [c.c.a.ApiServlet] (qtp681094281-318:ctx-fc79b118) (logid:8a349f6d) ===START=== 192.168.169.251 -- GET command=cloudianIsEnabled&response=json 2022-07-18 01:17:34,163 DEBUG [c.c.a.ApiServer] (qtp681094281-318:ctx-fc79b118 ctx-40fd8f3a) (logid:8a349f6d) CIDRs from which account 'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2, "name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is allowed to perform API calls: 0.0.0.0/0,::/0 2022-07-18 01:17:34,168 DEBUG [c.c.a.ApiServlet] (qtp681094281-318:ctx-fc79b118 ctx-40fd8f3a) (logid:8a349f6d) ===END=== 192.168.169.251 -- GET command=cloudianIsEnabled&response=json 2022-07-18 01:17:34,176 DEBUG [c.c.a.ApiServlet] (qtp681094281-34:ctx-20a51695) (logid:2436a576) ===START=== 192.168.12022-07-18 01:17:34,123 DEBUG [c.c.a.ApiServlet] (qtp681094281-280:ctx-fafe166c) (logid:41d7b4d5) ===START=== 192.168.169.251 -- GET listall=true&command=listZones&response=json 2022-07-18 01:17:34,133 DEBUG [c.c.a.ApiServer] (qtp681094281-280:ctx-fafe166c ctx-2269cc31) (logid:41d7b4d5) CIDRs from which account 'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2, "name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is allowed to perform API calls: 0.0.0.0/0,::/0 2022-07-18 01:17:34,133 DEBUG [c.c.a.ApiServlet] (qtp681094281-28:ctx-0906d03f) (logid:56b10f23) ===START=== 192.168.169.251 -- GET command=listApis&response=json 2022-07-18 01:17:34,137 DEBUG [c.c.a.ApiServlet] (qtp681094281-280:ctx-fafe166c ctx-2269cc31) (logid:41d7b4d5) ===END=== 192.168.169.251 -- GET listall=true&command=listZones&response=json 2022-07-18 01:17:34,144 DEBUG [c.c.a.ApiServer] (qtp681094281-28:ctx-0906d03f ctx-5a2a7dde) (logid:56b10f23) CIDRs from which account 'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2, "name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is allowed to perform API calls: 0.0.0.0/0,::/0 2022-07-18 01:17:34,153 DEBUG [c.c.a.ApiServlet] (qtp681094281-318:ctx-fc79b118) (logid:8a349f6d) ===START=== 192.168.169.251 -- GET command=cloudianIsEnabled&response=json 2022-07-18 01:17:34,163 DEBUG [c.c.a.ApiServer] (qtp681094281-318:ctx-fc79b118 ctx-40fd8f3a) (logid:8a349f6d) CIDRs from which account 'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2, "name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is allowed to perform API calls: 0.0.0.0/0,::/0 2022-07-18 01:17:34,168 DEBUG [c.c.a.ApiServlet] (qtp681094281-318:ctx-fc79b118 ctx-40fd8f3a) (logid:8a349f6d) ===END=== 192.168.169.251 -- GET command=cloudianIsEnabled&response=json 2022-07-18 01:17:34,176 DEBUG [c.c.a.ApiServlet] (qtp681094281-34:ctx-20a51695) (logid:2436a576) ===START=== 192.168.169.251 -- GET command=listLdapConfigurations&response=json 2022-07-18 01:17:34,185 DEBUG [c.c.a.ApiServer] (qtp681094281-34:ctx-20a51695 ctx-73e9ab8d) (logid:2436a576) CIDRs from which account 'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2, "name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is allowed to perform API calls: 0.0.0.0/0,::/0 2022-07-18 01:17:34,188 DEBUG [c.c.a.ApiServlet] (qtp681094281-34:ctx-20a51695 ctx-73e9ab8d) (logid:2436a576) ===END=== 192.168.169.251 -- GET command=listLdapConfigurations&response=json 2022-07-18 01:17:34,196 DEBUG [c.c.a.ApiServlet] (qtp681094281-343:ctx-43a80d6a) (logid:8d0a86c5) ===START=== 192.168.169.251 -- GET command=listCapabilities&response=json 2022-07-18 01:17:34,208 DEBUG [c.c.a.ApiServlet] (qtp681094281-343:ctx-43a80d6a ctx-dc6fb55f) (logid:8d0a86c5) ===END=== 192.168.169.251 -- GET command=listCapabilities&response=json 2022-07-18 01:17:34,218 DEBUG [c.c.a.ApiServlet] (qtp681094281-339:ctx-7d400edb) (logid:a57fa769) ===START=== 192.168.169.251 -- GET username=andrei&command=listUsers&response=json 2022-07-18 01:17:34,227 DEBUG [c.c.a.ApiServer] (qtp681094281-339:ctx-7d400edb ctx-2b12ac89) (logid:a57fa769) CIDRs from which account 'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2, "name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is allowed to perform API calls: 0.0.0.0/0,::/0 2022-07-18 01:17:34,230 DEBUG [c.c.a.ApiServlet] (qtp681094281-339:ctx-7d400edb ctx-2b12ac89) (logid:a57fa769) ===END=== 192.168.169.251 -- GET username=andrei&command=listUsers&response=json -------------- I can successfully login to the primary management server. I've done some further investigation from the client browser side to see what requests are being exchanged between the browser and the management server. It seems that the second management server gives me a bunch of 401 errors during the login session. There are some http 200 responses, but mainly 401For example: Client Request: POST /client/api/ HTTP/1.1 Server Response: HTTP/1.1 200 OK {"loginresponse":{"username":"andrei","userid":"ee8bbe57-acce-47fa-8d9b-9e831dcf87a2","domainid":"334d7527-65f1-11e3-9bd1-d8d38559b2d0","timeout":1800,"account":"admin_group","firstname":"Andrei","lastname":"Mikhailovsky","type":"1","timezone":"Etc/UTC","timezoneoffset":"0.0","registered":"false","sessionkey":"XXXX"}} ----- Client Request: GET /client/api/?listall=true&command=listZones&response=json HTTP/1.1 Server Response: HTTP/1.1 401 Unauthorized {"listzonesresponse":{"uuidList":[],"errorcode":401,"cserrorcode":9999,"errortext":"The given command 'listZones' either does not exist, is not available for user."}} ----- Client Request: GET /client/api/?command=listApis&response=json HTTP/1.1 Server Response: HTTP/1.1 200 OK {"listapisresponse":{"count":96,"api":[{"name":"listResourceIcon","description":"Lists the resource icon for the specified resource(s)","since":"4.16.0.0","isasync":false,"related":"","params":[{"name":"resourcetype","description":"type of the resource","type":"string","length":255,"required":true}, (Followed by about 200K other data in the above request) ----- Client Requests: GET /client/api/?username=andrei&command=listUsers&response=json HTTP/1.1 GET /client/api/?command=listLdapConfigurations&response=json HTTP/1.1 GET /client/api/?command=listCapabilities&response=json HTTP/1.1 Server Response (for the above 3 requests): HTTP/1.1 401 Unauthorized {"listusersresponse":{"uuidList":[],"errorcode":401,"cserrorcode":9999,"errortext":"The given command 'listUsers' either does not exist, is not available for user."}} ---------------- Does anyone know what could be causing the login issues on the second management server? How do I solve the issue? Many thanks