Sam Corbett created BROOKLYN-167:
------------------------------------
Summary: HighAvailabilityManagerImpl has invalid state if
/v1/server/ha/states requested before BrooklynLauncher has fully initialised a
server
Key: BROOKLYN-167
URL: https://issues.apache.org/jira/browse/BROOKLYN-167
Project: Brooklyn
Issue Type: Bug
Reporter: Sam Corbett
If requests are made to get a server's HA state before the server is fully
initialised the HA state ends up incorrect.
I started Brooklyn with this command:
{{brooklyn launch --persist disabled}}
I happened to have the jsgui open in my browser from a previous session.
The HA manager appeared to enter an invalid state:
{code}
sam@v:~/code/brooklyn.io/brooklyn (master $)
$ http :8081/v1/server/ha/state
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store
Content-Encoding: gzip
Content-Type: application/json
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Expires: 0
Pragma: no-cache
Server: Jetty(8.1.17.v20150415)
Set-Cookie: JSESSIONID=12nhcq9l0tmll1abkqmaorn3lp;Path=/
Transfer-Encoding: chunked
Vary: Accept-Encoding
"MASTER"
sam@v:~/code/brooklyn.io/brooklyn (master $)
$ http :8081/v1/server/ha/states
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store
Content-Encoding: gzip
Content-Type: application/json
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Expires: 0
Pragma: no-cache
Server: Jetty(8.1.17.v20150415)
Set-Cookie: JSESSIONID=8u4e6sv9f6h2g7xee0vsym92;Path=/
Transfer-Encoding: chunked
Vary: Accept-Encoding
{
"links": {},
"masterId": null,
"nodes": {},
"ownId": "iuIF5Nlr"
}
{code}
The debug log (attached) contains the following messages:
{code}
2015-08-24 11:14:52,433 DEBUG o.a.b.l.BrooklynLauncher [main]: Starting
Brooklyn using security properties: BrooklynProperties[1]
2015-08-24 11:14:52,433 DEBUG o.a.b.l.BrooklynLauncher [main]: Starting
Brooklyn web-console with bindAddress 0.0.0.0/0.0.0.0 and properties
BrooklynProperties[480]
2015-08-24 11:14:52,829 DEBUG o.a.b.l.BrooklynWebServer [main]: Starting
Brooklyn console at http://127.0.0.1:8081/, running classpath://brooklyn.war
and []
2015-08-24 11:14:53,363 INFO c.s.j.s.i.a.WebApplicationImpl [main]: Initiating
Jersey application, version 'Jersey: 1.18.1 02/19/2014 03:28 AM'
2015-08-24 11:14:54,405 INFO o.a.b.r.s.p.DelegatingSecurityProvider
[brooklyn-jetty-server-8081-qtp1643988252-22]: REST using security provider
org.apache.brooklyn.rest.security.provider.AnyoneSecurityProvider
2015-08-24 11:14:54,426 WARN o.a.b.r.f.HaHotCheckResourceFilter
[brooklyn-jetty-server-8081-qtp1643988252-21]: Disallowing web request as
server not in required startup-completed state:
com.sun.jersey.spi.container.ContainerRequest@7c320b37/AbstractResourceMethod(LocationApi#list)
(caller should set 'Brooklyn-Allow-Non-Master-Access' to force)
2015-08-24 11:14:54,426 WARN o.a.b.r.f.HaHotCheckResourceFilter
[brooklyn-jetty-server-8081-qtp1643988252-22]: Disallowing web request as
server not in required startup-completed state:
com.sun.jersey.spi.container.ContainerRequest@5595c2ac/AbstractResourceMethod(ApplicationApi#list)
(caller should set 'Brooklyn-Allow-Non-Master-Access' to force)
2015-08-24 11:14:54,427 DEBUG o.a.b.c.m.h.HighAvailabilityManagerImpl
[brooklyn-jetty-server-8081-qtp1643988252-23]: High availablity manager has no
persister; returning empty record
...
2015-08-24 11:14:54,602 INFO o.a.b.l.BrooklynLauncher [main]: Persistence
disabled
2015-08-24 11:14:54,602 INFO o.a.b.l.BrooklynLauncher [main]: High
availability disabled
2015-08-24 11:14:54,602 DEBUG o.a.b.l.BrooklynLauncher [main]: Loading catalog
as part of launch sequence (it was not loaded as part of any rebind sequence)
{code}
The message "High availablity manager has no persister" is logged by
HighAvailabilityManagerImpl in this code:
{code:java}
private ManagementPlaneSyncRecord
loadManagementPlaneSyncRecordInternal(boolean useLocalKnowledgeForThisNode) {
if (disabled) {
// if HA is disabled, then we are the only node - no persistence;
just load a memento to describe this node
Builder builder = ManagementPlaneSyncRecordImpl.builder()
.node(createManagementNodeSyncRecord(true));
if (getTransitionTargetNodeState() == ManagementNodeState.MASTER) {
builder.masterNodeId(ownNodeId);
}
return builder.build();
}
if (persister == null) {
// e.g. web-console may be polling before we've started up
LOG.debug("High availablity manager has no persister; returning
empty record");
return ManagementPlaneSyncRecordImpl.builder().build();
}
...
}
{code}
The first HTTP request occurs before BrooklynLauncher indicates to
HighAvailabilityManagerImpl that persistence is disabled and Brooklyn gets
stuck in an invalid state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
