[
https://issues.apache.org/jira/browse/BROOKLYN-167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Corbett updated BROOKLYN-167:
---------------------------------
Description:
If requests are made to get a server's HA state before the server is fully
initialised the HA state ends up incorrect.
I started Brooklyn with this command:
{{brooklyn launch --persist disabled}}
I happened to have the jsgui open in my browser from a previous session.
The HA manager appeared to enter an invalid state:
{code}
sam@v:~/code/brooklyn.io/brooklyn (master $)
$ http :8081/v1/server/ha/state
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store
Content-Encoding: gzip
Content-Type: application/json
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Expires: 0
Pragma: no-cache
Server: Jetty(8.1.17.v20150415)
Set-Cookie: JSESSIONID=12nhcq9l0tmll1abkqmaorn3lp;Path=/
Transfer-Encoding: chunked
Vary: Accept-Encoding
"MASTER"
sam@v:~/code/brooklyn.io/brooklyn (master $)
$ http :8081/v1/server/ha/states
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store
Content-Encoding: gzip
Content-Type: application/json
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Expires: 0
Pragma: no-cache
Server: Jetty(8.1.17.v20150415)
Set-Cookie: JSESSIONID=8u4e6sv9f6h2g7xee0vsym92;Path=/
Transfer-Encoding: chunked
Vary: Accept-Encoding
{
"links": {},
"masterId": null,
"nodes": {},
"ownId": "iuIF5Nlr"
}
{code}
The debug log contains the following messages:
{code}
2015-08-24 11:14:52,433 DEBUG o.a.b.l.BrooklynLauncher [main]: Starting
Brooklyn using security properties: BrooklynProperties[1]
2015-08-24 11:14:52,433 DEBUG o.a.b.l.BrooklynLauncher [main]: Starting
Brooklyn web-console with bindAddress 0.0.0.0/0.0.0.0 and properties
BrooklynProperties[480]
2015-08-24 11:14:52,829 DEBUG o.a.b.l.BrooklynWebServer [main]: Starting
Brooklyn console at http://127.0.0.1:8081/, running classpath://brooklyn.war
and []
2015-08-24 11:14:53,363 INFO c.s.j.s.i.a.WebApplicationImpl [main]: Initiating
Jersey application, version 'Jersey: 1.18.1 02/19/2014 03:28 AM'
2015-08-24 11:14:54,405 INFO o.a.b.r.s.p.DelegatingSecurityProvider
[brooklyn-jetty-server-8081-qtp1643988252-22]: REST using security provider
org.apache.brooklyn.rest.security.provider.AnyoneSecurityProvider
2015-08-24 11:14:54,426 WARN o.a.b.r.f.HaHotCheckResourceFilter
[brooklyn-jetty-server-8081-qtp1643988252-21]: Disallowing web request as
server not in required startup-completed state:
com.sun.jersey.spi.container.ContainerRequest@7c320b37/AbstractResourceMethod(LocationApi#list)
(caller should set 'Brooklyn-Allow-Non-Master-Access' to force)
2015-08-24 11:14:54,426 WARN o.a.b.r.f.HaHotCheckResourceFilter
[brooklyn-jetty-server-8081-qtp1643988252-22]: Disallowing web request as
server not in required startup-completed state:
com.sun.jersey.spi.container.ContainerRequest@5595c2ac/AbstractResourceMethod(ApplicationApi#list)
(caller should set 'Brooklyn-Allow-Non-Master-Access' to force)
2015-08-24 11:14:54,427 DEBUG o.a.b.c.m.h.HighAvailabilityManagerImpl
[brooklyn-jetty-server-8081-qtp1643988252-23]: High availablity manager has no
persister; returning empty record
...
2015-08-24 11:14:54,602 INFO o.a.b.l.BrooklynLauncher [main]: Persistence
disabled
2015-08-24 11:14:54,602 INFO o.a.b.l.BrooklynLauncher [main]: High
availability disabled
2015-08-24 11:14:54,602 DEBUG o.a.b.l.BrooklynLauncher [main]: Loading catalog
as part of launch sequence (it was not loaded as part of any rebind sequence)
{code}
The message "High availablity manager has no persister" is logged by
HighAvailabilityManagerImpl in this code:
{code:java}
private ManagementPlaneSyncRecord
loadManagementPlaneSyncRecordInternal(boolean useLocalKnowledgeForThisNode) {
if (disabled) {
// if HA is disabled, then we are the only node - no persistence;
just load a memento to describe this node
Builder builder = ManagementPlaneSyncRecordImpl.builder()
.node(createManagementNodeSyncRecord(true));
if (getTransitionTargetNodeState() == ManagementNodeState.MASTER) {
builder.masterNodeId(ownNodeId);
}
return builder.build();
}
if (persister == null) {
// e.g. web-console may be polling before we've started up
LOG.debug("High availablity manager has no persister; returning
empty record");
return ManagementPlaneSyncRecordImpl.builder().build();
}
...
}
{code}
The first HTTP request occurs before BrooklynLauncher indicates to
HighAvailabilityManagerImpl that persistence is disabled and Brooklyn gets
stuck in an invalid state.
was:
If requests are made to get a server's HA state before the server is fully
initialised the HA state ends up incorrect.
I started Brooklyn with this command:
{{brooklyn launch --persist disabled}}
I happened to have the jsgui open in my browser from a previous session.
The HA manager appeared to enter an invalid state:
{code}
sam@v:~/code/brooklyn.io/brooklyn (master $)
$ http :8081/v1/server/ha/state
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store
Content-Encoding: gzip
Content-Type: application/json
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Expires: 0
Pragma: no-cache
Server: Jetty(8.1.17.v20150415)
Set-Cookie: JSESSIONID=12nhcq9l0tmll1abkqmaorn3lp;Path=/
Transfer-Encoding: chunked
Vary: Accept-Encoding
"MASTER"
sam@v:~/code/brooklyn.io/brooklyn (master $)
$ http :8081/v1/server/ha/states
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store
Content-Encoding: gzip
Content-Type: application/json
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Expires: 0
Pragma: no-cache
Server: Jetty(8.1.17.v20150415)
Set-Cookie: JSESSIONID=8u4e6sv9f6h2g7xee0vsym92;Path=/
Transfer-Encoding: chunked
Vary: Accept-Encoding
{
"links": {},
"masterId": null,
"nodes": {},
"ownId": "iuIF5Nlr"
}
{code}
The debug log (attached) contains the following messages:
{code}
2015-08-24 11:14:52,433 DEBUG o.a.b.l.BrooklynLauncher [main]: Starting
Brooklyn using security properties: BrooklynProperties[1]
2015-08-24 11:14:52,433 DEBUG o.a.b.l.BrooklynLauncher [main]: Starting
Brooklyn web-console with bindAddress 0.0.0.0/0.0.0.0 and properties
BrooklynProperties[480]
2015-08-24 11:14:52,829 DEBUG o.a.b.l.BrooklynWebServer [main]: Starting
Brooklyn console at http://127.0.0.1:8081/, running classpath://brooklyn.war
and []
2015-08-24 11:14:53,363 INFO c.s.j.s.i.a.WebApplicationImpl [main]: Initiating
Jersey application, version 'Jersey: 1.18.1 02/19/2014 03:28 AM'
2015-08-24 11:14:54,405 INFO o.a.b.r.s.p.DelegatingSecurityProvider
[brooklyn-jetty-server-8081-qtp1643988252-22]: REST using security provider
org.apache.brooklyn.rest.security.provider.AnyoneSecurityProvider
2015-08-24 11:14:54,426 WARN o.a.b.r.f.HaHotCheckResourceFilter
[brooklyn-jetty-server-8081-qtp1643988252-21]: Disallowing web request as
server not in required startup-completed state:
com.sun.jersey.spi.container.ContainerRequest@7c320b37/AbstractResourceMethod(LocationApi#list)
(caller should set 'Brooklyn-Allow-Non-Master-Access' to force)
2015-08-24 11:14:54,426 WARN o.a.b.r.f.HaHotCheckResourceFilter
[brooklyn-jetty-server-8081-qtp1643988252-22]: Disallowing web request as
server not in required startup-completed state:
com.sun.jersey.spi.container.ContainerRequest@5595c2ac/AbstractResourceMethod(ApplicationApi#list)
(caller should set 'Brooklyn-Allow-Non-Master-Access' to force)
2015-08-24 11:14:54,427 DEBUG o.a.b.c.m.h.HighAvailabilityManagerImpl
[brooklyn-jetty-server-8081-qtp1643988252-23]: High availablity manager has no
persister; returning empty record
...
2015-08-24 11:14:54,602 INFO o.a.b.l.BrooklynLauncher [main]: Persistence
disabled
2015-08-24 11:14:54,602 INFO o.a.b.l.BrooklynLauncher [main]: High
availability disabled
2015-08-24 11:14:54,602 DEBUG o.a.b.l.BrooklynLauncher [main]: Loading catalog
as part of launch sequence (it was not loaded as part of any rebind sequence)
{code}
The message "High availablity manager has no persister" is logged by
HighAvailabilityManagerImpl in this code:
{code:java}
private ManagementPlaneSyncRecord
loadManagementPlaneSyncRecordInternal(boolean useLocalKnowledgeForThisNode) {
if (disabled) {
// if HA is disabled, then we are the only node - no persistence;
just load a memento to describe this node
Builder builder = ManagementPlaneSyncRecordImpl.builder()
.node(createManagementNodeSyncRecord(true));
if (getTransitionTargetNodeState() == ManagementNodeState.MASTER) {
builder.masterNodeId(ownNodeId);
}
return builder.build();
}
if (persister == null) {
// e.g. web-console may be polling before we've started up
LOG.debug("High availablity manager has no persister; returning
empty record");
return ManagementPlaneSyncRecordImpl.builder().build();
}
...
}
{code}
The first HTTP request occurs before BrooklynLauncher indicates to
HighAvailabilityManagerImpl that persistence is disabled and Brooklyn gets
stuck in an invalid state.
> HighAvailabilityManagerImpl has invalid state if /v1/server/ha/states
> requested before BrooklynLauncher has fully initialised a server
> --------------------------------------------------------------------------------------------------------------------------------------
>
> Key: BROOKLYN-167
> URL: https://issues.apache.org/jira/browse/BROOKLYN-167
> Project: Brooklyn
> Issue Type: Bug
> Reporter: Sam Corbett
>
> If requests are made to get a server's HA state before the server is fully
> initialised the HA state ends up incorrect.
> I started Brooklyn with this command:
> {{brooklyn launch --persist disabled}}
> I happened to have the jsgui open in my browser from a previous session.
> The HA manager appeared to enter an invalid state:
> {code}
> sam@v:~/code/brooklyn.io/brooklyn (master $)
> $ http :8081/v1/server/ha/state
> HTTP/1.1 200 OK
> Cache-Control: no-cache, no-store
> Content-Encoding: gzip
> Content-Type: application/json
> Expires: Thu, 01 Jan 1970 00:00:00 GMT
> Expires: 0
> Pragma: no-cache
> Server: Jetty(8.1.17.v20150415)
> Set-Cookie: JSESSIONID=12nhcq9l0tmll1abkqmaorn3lp;Path=/
> Transfer-Encoding: chunked
> Vary: Accept-Encoding
> "MASTER"
> sam@v:~/code/brooklyn.io/brooklyn (master $)
> $ http :8081/v1/server/ha/states
> HTTP/1.1 200 OK
> Cache-Control: no-cache, no-store
> Content-Encoding: gzip
> Content-Type: application/json
> Expires: Thu, 01 Jan 1970 00:00:00 GMT
> Expires: 0
> Pragma: no-cache
> Server: Jetty(8.1.17.v20150415)
> Set-Cookie: JSESSIONID=8u4e6sv9f6h2g7xee0vsym92;Path=/
> Transfer-Encoding: chunked
> Vary: Accept-Encoding
> {
> "links": {},
> "masterId": null,
> "nodes": {},
> "ownId": "iuIF5Nlr"
> }
> {code}
> The debug log contains the following messages:
> {code}
> 2015-08-24 11:14:52,433 DEBUG o.a.b.l.BrooklynLauncher [main]: Starting
> Brooklyn using security properties: BrooklynProperties[1]
> 2015-08-24 11:14:52,433 DEBUG o.a.b.l.BrooklynLauncher [main]: Starting
> Brooklyn web-console with bindAddress 0.0.0.0/0.0.0.0 and properties
> BrooklynProperties[480]
> 2015-08-24 11:14:52,829 DEBUG o.a.b.l.BrooklynWebServer [main]: Starting
> Brooklyn console at http://127.0.0.1:8081/, running classpath://brooklyn.war
> and []
> 2015-08-24 11:14:53,363 INFO c.s.j.s.i.a.WebApplicationImpl [main]:
> Initiating Jersey application, version 'Jersey: 1.18.1 02/19/2014 03:28 AM'
> 2015-08-24 11:14:54,405 INFO o.a.b.r.s.p.DelegatingSecurityProvider
> [brooklyn-jetty-server-8081-qtp1643988252-22]: REST using security provider
> org.apache.brooklyn.rest.security.provider.AnyoneSecurityProvider
> 2015-08-24 11:14:54,426 WARN o.a.b.r.f.HaHotCheckResourceFilter
> [brooklyn-jetty-server-8081-qtp1643988252-21]: Disallowing web request as
> server not in required startup-completed state:
> com.sun.jersey.spi.container.ContainerRequest@7c320b37/AbstractResourceMethod(LocationApi#list)
> (caller should set 'Brooklyn-Allow-Non-Master-Access' to force)
> 2015-08-24 11:14:54,426 WARN o.a.b.r.f.HaHotCheckResourceFilter
> [brooklyn-jetty-server-8081-qtp1643988252-22]: Disallowing web request as
> server not in required startup-completed state:
> com.sun.jersey.spi.container.ContainerRequest@5595c2ac/AbstractResourceMethod(ApplicationApi#list)
> (caller should set 'Brooklyn-Allow-Non-Master-Access' to force)
> 2015-08-24 11:14:54,427 DEBUG o.a.b.c.m.h.HighAvailabilityManagerImpl
> [brooklyn-jetty-server-8081-qtp1643988252-23]: High availablity manager has
> no persister; returning empty record
> ...
> 2015-08-24 11:14:54,602 INFO o.a.b.l.BrooklynLauncher [main]: Persistence
> disabled
> 2015-08-24 11:14:54,602 INFO o.a.b.l.BrooklynLauncher [main]: High
> availability disabled
> 2015-08-24 11:14:54,602 DEBUG o.a.b.l.BrooklynLauncher [main]: Loading
> catalog as part of launch sequence (it was not loaded as part of any rebind
> sequence)
> {code}
> The message "High availablity manager has no persister" is logged by
> HighAvailabilityManagerImpl in this code:
> {code:java}
> private ManagementPlaneSyncRecord
> loadManagementPlaneSyncRecordInternal(boolean useLocalKnowledgeForThisNode) {
> if (disabled) {
> // if HA is disabled, then we are the only node - no persistence;
> just load a memento to describe this node
> Builder builder = ManagementPlaneSyncRecordImpl.builder()
> .node(createManagementNodeSyncRecord(true));
> if (getTransitionTargetNodeState() == ManagementNodeState.MASTER)
> {
> builder.masterNodeId(ownNodeId);
> }
> return builder.build();
> }
> if (persister == null) {
> // e.g. web-console may be polling before we've started up
> LOG.debug("High availablity manager has no persister; returning
> empty record");
> return ManagementPlaneSyncRecordImpl.builder().build();
> }
> ...
> }
> {code}
> The first HTTP request occurs before BrooklynLauncher indicates to
> HighAvailabilityManagerImpl that persistence is disabled and Brooklyn gets
> stuck in an invalid state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
