Hello,
I've recently installed a second management server ACS 4.16.1 following the
installation instructions in section Additional Management Servers from the
official documentation ( [
http://docs.cloudstack.apache.org/en/4.16.1.0/installguide/management-server/index.html
|
http://docs.cloudstack.apache.org/en/4.16.1.0/installguide/management-server/index.html
] ). I've installed the Ubuntu package on the second server of the same
version as the primary management server. Configured the database with
cloudstack-setup-databases command followed by running
cloudstack-setup-management as per the documentation. There were no errors in
the process and the cloudstack-management.service seems to have started just
fine. The second ACS management service connected to the same database as the
primary one and the login web GUI loaded just fine. The management server logs
seems to show no apparent errors in the startup. The only exceptions I was
getting in the logs were from the host agents showing status Disconnected.
So, I have tried to login (using domain and ROOT login accounts) to the web gui
of the second management server and the page just hangs after I enter the
credentials and press the Login button. I've tried several different browsers
at no avail. Supplying the incorrect login credentials produce the error
though. The management server logs do not show any errors during the login
process. In fact, it seems that all commands produce " is allowed to perform
API calls: 0.0.0.0/0,::/0 " message in the logs. There are no exceptions that I
can see either:
--------------
2022-07-18 01:17:33,743 DEBUG [c.c.a.ApiServlet]
(qtp681094281-285:ctx-0cf08734) (logid:94b277ba) ===START=== 192.168.169.251 --
POST
2022-07-18 01:17:33,750 DEBUG [c.c.u.AccountManagerImpl]
(qtp681094281-285:ctx-0cf08734) (logid:94b277ba) Attempting to log in user:
andrei in domain 1
2022-07-18 01:17:33,752 DEBUG [o.a.c.s.a.PBKDF2UserAuthenticator]
(qtp681094281-285:ctx-0cf08734) (logid:94b277ba) Retrieving user: andrei
2022-07-18 01:17:33,969 DEBUG [c.c.u.AccountManagerImpl]
(qtp681094281-285:ctx-0cf08734) (logid:94b277ba) CIDRs from which account
'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2,
"name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is
allowed to perform API calls: 0.0.0.0/0,::/0
2022-07-18 01:17:33,969 DEBUG [c.c.u.AccountManagerImpl]
(qtp681094281-285:ctx-0cf08734) (logid:94b277ba) User: andrei in domain 1 has
successfully logged in
2022-07-18 01:17:34,011 INFO [c.c.a.ApiServer] (qtp681094281-285:ctx-0cf08734)
(logid:94b277ba) Current user logged in under Etc/UTC timezone
2022-07-18 01:17:34,011 INFO [c.c.a.ApiServer] (qtp681094281-285:ctx-0cf08734)
(logid:94b277ba) Timezone offset from UTC is: 0.0
2022-07-18 01:17:34,015 DEBUG [c.c.a.ApiServlet]
(qtp681094281-285:ctx-0cf08734) (logid:94b277ba) ===END=== 192.168.169.251 --
POST
2022-07-18 01:17:34,123 DEBUG [c.c.a.ApiServlet]
(qtp681094281-280:ctx-fafe166c) (logid:41d7b4d5) ===START=== 192.168.169.251 --
GET listall=true&command=listZones&response=json
2022-07-18 01:17:34,133 DEBUG [c.c.a.ApiServer] (qtp681094281-280:ctx-fafe166c
ctx-2269cc31) (logid:41d7b4d5) CIDRs from which account
'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2,
"name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is
allowed to perform API calls: 0.0.0.0/0,::/0
2022-07-18 01:17:34,133 DEBUG [c.c.a.ApiServlet] (qtp681094281-28:ctx-0906d03f)
(logid:56b10f23) ===START=== 192.168.169.251 -- GET
command=listApis&response=json
2022-07-18 01:17:34,137 DEBUG [c.c.a.ApiServlet] (qtp681094281-280:ctx-fafe166c
ctx-2269cc31) (logid:41d7b4d5) ===END=== 192.168.169.251 -- GET
listall=true&command=listZones&response=json
2022-07-18 01:17:34,144 DEBUG [c.c.a.ApiServer] (qtp681094281-28:ctx-0906d03f
ctx-5a2a7dde) (logid:56b10f23) CIDRs from which account
'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2,
"name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is
allowed to perform API calls: 0.0.0.0/0,::/0
2022-07-18 01:17:34,153 DEBUG [c.c.a.ApiServlet]
(qtp681094281-318:ctx-fc79b118) (logid:8a349f6d) ===START=== 192.168.169.251 --
GET command=cloudianIsEnabled&response=json
2022-07-18 01:17:34,163 DEBUG [c.c.a.ApiServer] (qtp681094281-318:ctx-fc79b118
ctx-40fd8f3a) (logid:8a349f6d) CIDRs from which account
'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2,
"name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is
allowed to perform API calls: 0.0.0.0/0,::/0
2022-07-18 01:17:34,168 DEBUG [c.c.a.ApiServlet] (qtp681094281-318:ctx-fc79b118
ctx-40fd8f3a) (logid:8a349f6d) ===END=== 192.168.169.251 -- GET
command=cloudianIsEnabled&response=json
2022-07-18 01:17:34,176 DEBUG [c.c.a.ApiServlet] (qtp681094281-34:ctx-20a51695)
(logid:2436a576) ===START=== 192.168.12022-07-18 01:17:34,123 DEBUG
[c.c.a.ApiServlet] (qtp681094281-280:ctx-fafe166c) (logid:41d7b4d5) ===START===
192.168.169.251 -- GET listall=true&command=listZones&response=json
2022-07-18 01:17:34,133 DEBUG [c.c.a.ApiServer] (qtp681094281-280:ctx-fafe166c
ctx-2269cc31) (logid:41d7b4d5) CIDRs from which account
'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2,
"name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is
allowed to perform API calls: 0.0.0.0/0,::/0
2022-07-18 01:17:34,133 DEBUG [c.c.a.ApiServlet] (qtp681094281-28:ctx-0906d03f)
(logid:56b10f23) ===START=== 192.168.169.251 -- GET
command=listApis&response=json
2022-07-18 01:17:34,137 DEBUG [c.c.a.ApiServlet] (qtp681094281-280:ctx-fafe166c
ctx-2269cc31) (logid:41d7b4d5) ===END=== 192.168.169.251 -- GET
listall=true&command=listZones&response=json
2022-07-18 01:17:34,144 DEBUG [c.c.a.ApiServer] (qtp681094281-28:ctx-0906d03f
ctx-5a2a7dde) (logid:56b10f23) CIDRs from which account
'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2,
"name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is
allowed to perform API calls: 0.0.0.0/0,::/0
2022-07-18 01:17:34,153 DEBUG [c.c.a.ApiServlet]
(qtp681094281-318:ctx-fc79b118) (logid:8a349f6d) ===START=== 192.168.169.251 --
GET command=cloudianIsEnabled&response=json
2022-07-18 01:17:34,163 DEBUG [c.c.a.ApiServer] (qtp681094281-318:ctx-fc79b118
ctx-40fd8f3a) (logid:8a349f6d) CIDRs from which account
'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2,
"name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is
allowed to perform API calls: 0.0.0.0/0,::/0
2022-07-18 01:17:34,168 DEBUG [c.c.a.ApiServlet] (qtp681094281-318:ctx-fc79b118
ctx-40fd8f3a) (logid:8a349f6d) ===END=== 192.168.169.251 -- GET
command=cloudianIsEnabled&response=json
2022-07-18 01:17:34,176 DEBUG [c.c.a.ApiServlet] (qtp681094281-34:ctx-20a51695)
(logid:2436a576) ===START=== 192.168.12022-07-18 01:17:34,123 DEBUG
[c.c.a.ApiServlet] (qtp681094281-280:ctx-fafe166c) (logid:41d7b4d5) ===START===
192.168.169.251 -- GET listall=true&command=listZones&response=json
2022-07-18 01:17:34,133 DEBUG [c.c.a.ApiServer] (qtp681094281-280:ctx-fafe166c
ctx-2269cc31) (logid:41d7b4d5) CIDRs from which account
'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2,
"name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is
allowed to perform API calls: 0.0.0.0/0,::/0
2022-07-18 01:17:34,133 DEBUG [c.c.a.ApiServlet] (qtp681094281-28:ctx-0906d03f)
(logid:56b10f23) ===START=== 192.168.169.251 -- GET
command=listApis&response=json
2022-07-18 01:17:34,137 DEBUG [c.c.a.ApiServlet] (qtp681094281-280:ctx-fafe166c
ctx-2269cc31) (logid:41d7b4d5) ===END=== 192.168.169.251 -- GET
listall=true&command=listZones&response=json
2022-07-18 01:17:34,144 DEBUG [c.c.a.ApiServer] (qtp681094281-28:ctx-0906d03f
ctx-5a2a7dde) (logid:56b10f23) CIDRs from which account
'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2,
"name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is
allowed to perform API calls: 0.0.0.0/0,::/0
2022-07-18 01:17:34,153 DEBUG [c.c.a.ApiServlet]
(qtp681094281-318:ctx-fc79b118) (logid:8a349f6d) ===START=== 192.168.169.251 --
GET command=cloudianIsEnabled&response=json
2022-07-18 01:17:34,163 DEBUG [c.c.a.ApiServer] (qtp681094281-318:ctx-fc79b118
ctx-40fd8f3a) (logid:8a349f6d) CIDRs from which account
'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2,
"name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is
allowed to perform API calls: 0.0.0.0/0,::/0
2022-07-18 01:17:34,168 DEBUG [c.c.a.ApiServlet] (qtp681094281-318:ctx-fc79b118
ctx-40fd8f3a) (logid:8a349f6d) ===END=== 192.168.169.251 -- GET
command=cloudianIsEnabled&response=json
2022-07-18 01:17:34,176 DEBUG [c.c.a.ApiServlet] (qtp681094281-34:ctx-20a51695)
(logid:2436a576) ===START=== 192.168.169.251 -- GET
command=listLdapConfigurations&response=json
2022-07-18 01:17:34,185 DEBUG [c.c.a.ApiServer] (qtp681094281-34:ctx-20a51695
ctx-73e9ab8d) (logid:2436a576) CIDRs from which account
'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2,
"name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is
allowed to perform API calls: 0.0.0.0/0,::/0
2022-07-18 01:17:34,188 DEBUG [c.c.a.ApiServlet] (qtp681094281-34:ctx-20a51695
ctx-73e9ab8d) (logid:2436a576) ===END=== 192.168.169.251 -- GET
command=listLdapConfigurations&response=json
2022-07-18 01:17:34,196 DEBUG [c.c.a.ApiServlet]
(qtp681094281-343:ctx-43a80d6a) (logid:8d0a86c5) ===START=== 192.168.169.251 --
GET command=listCapabilities&response=json
2022-07-18 01:17:34,208 DEBUG [c.c.a.ApiServlet] (qtp681094281-343:ctx-43a80d6a
ctx-dc6fb55f) (logid:8d0a86c5) ===END=== 192.168.169.251 -- GET
command=listCapabilities&response=json
2022-07-18 01:17:34,218 DEBUG [c.c.a.ApiServlet]
(qtp681094281-339:ctx-7d400edb) (logid:a57fa769) ===START=== 192.168.169.251 --
GET username=andrei&command=listUsers&response=json
2022-07-18 01:17:34,227 DEBUG [c.c.a.ApiServer] (qtp681094281-339:ctx-7d400edb
ctx-2b12ac89) (logid:a57fa769) CIDRs from which account
'Acct[06eedc2c-65f2-11e3-9bd1-d8d38559b2d0-admin_group] -- Account {"id": 2,
"name": "admin_group", "uuid": "06eedc2c-65f2-11e3-9bd1-d8d38559b2d0"}' is
allowed to perform API calls: 0.0.0.0/0,::/0
2022-07-18 01:17:34,230 DEBUG [c.c.a.ApiServlet] (qtp681094281-339:ctx-7d400edb
ctx-2b12ac89) (logid:a57fa769) ===END=== 192.168.169.251 -- GET
username=andrei&command=listUsers&response=json
--------------
I can successfully login to the primary management server. I've done some
further investigation from the client browser side to see what requests are
being exchanged between the browser and the management server. It seems that
the second management server gives me a bunch of 401 errors during the login
session. There are some http 200 responses, but mainly 401For example:
Client Request:
POST /client/api/ HTTP/1.1
Server Response:
HTTP/1.1 200 OK
{"loginresponse":{"username":"andrei","userid":"ee8bbe57-acce-47fa-8d9b-9e831dcf87a2","domainid":"334d7527-65f1-11e3-9bd1-d8d38559b2d0","timeout":1800,"account":"admin_group","firstname":"Andrei","lastname":"Mikhailovsky","type":"1","timezone":"Etc/UTC","timezoneoffset":"0.0","registered":"false","sessionkey":"XXXX"}}
-----
Client Request:
GET /client/api/?listall=true&command=listZones&response=json HTTP/1.1
Server Response:
HTTP/1.1 401 Unauthorized
{"listzonesresponse":{"uuidList":[],"errorcode":401,"cserrorcode":9999,"errortext":"The
given command 'listZones' either does not exist, is not available for user."}}
-----
Client Request:
GET /client/api/?command=listApis&response=json HTTP/1.1
Server Response:
HTTP/1.1 200 OK
{"listapisresponse":{"count":96,"api":[{"name":"listResourceIcon","description":"Lists
the resource icon for the specified
resource(s)","since":"4.16.0.0","isasync":false,"related":"","params":[{"name":"resourcetype","description":"type
of the resource","type":"string","length":255,"required":true},
(Followed by about 200K other data in the above request)
-----
Client Requests:
GET /client/api/?username=andrei&command=listUsers&response=json HTTP/1.1
GET /client/api/?command=listLdapConfigurations&response=json HTTP/1.1
GET /client/api/?command=listCapabilities&response=json HTTP/1.1
Server Response (for the above 3 requests):
HTTP/1.1 401 Unauthorized
{"listusersresponse":{"uuidList":[],"errorcode":401,"cserrorcode":9999,"errortext":"The
given command 'listUsers' either does not exist, is not available for user."}}
----------------
Does anyone know what could be causing the login issues on the second
management server? How do I solve the issue?
Many thanks
