Nice catch Raj!
I think better to wait until contexts/policies are loaded from registry
before starting AutoscalerTopologyReceiver. We're seeing this issue now
because topology is initialized immediately after starting the receiver
with topology initializing optimization.
Thanks.
On Fri, Nov 13, 2015 at 10:23 AM, Rajkumar Rajaratnam <rajkum...@wso2.com>
wrote:
> Hi Devs,
>
> Experiencing $Subject.
>
> Root cause is, AS received the complete-topology-event before loading
> application-contexts from registry to in-memory model. So the
> complete-topology-event is ignored by AS, but the topology is initialized.
> This also means that all the upcoming periodic complete-topology-events
> will be ignored by AS. Refer the following code block in
> AutoscalerTopologyEventReceiver.
>
> private void addEventListeners() {
> // Listen to topology events that affect clusters
> topologyEventReceiver.addEventListener(new
> CompleteTopologyEventListener() {
> @Override
> protected void onEvent(Event event) {
> if (!topologyInitialized) {
> log.info("[CompleteTopologyEvent] Received: " +
> event.getClass());
> try {
> ApplicationHolder.acquireReadLock();
> Applications applications =
> ApplicationHolder.getApplications();
> if (applications != null) {
> for (Application application : applications.
> getApplications().values()) {
> ApplicationContext applicationContext =
> AutoscalerContext.getInstance().
>
> getApplicationContext(application.getUniqueIdentifier());
> if (applicationContext != null &&
> applicationContext.getStatus().
>
> equals(ApplicationContext.STATUS_DEPLOYED)) {
> if
> (AutoscalerUtil.allClustersInitialized(application)) {
>
> AutoscalerUtil.getInstance().startApplicationMonitor(
>
> application.getUniqueIdentifier());
> } else {
> log.error("Complete Topology is not
> consistent with " +
> "the applications which got
> persisted");
> }
> } else {
> log.info("The application is not yet " +
> "deployed for this [application] " +
> application.getUniqueIdentifier());
> }
>
> }
> topologyInitialized = true;
> } else {
> log.info("No applications found in the complete
> topology");
> }
> } catch (Exception e) {
> log.error("Error processing event", e);
> } finally {
> ApplicationHolder.releaseReadLock();
> }
> }
> }
> });
>
>
> This code block should ensure that the monitors should be created for all
> the persisted applications. Currently, it is not ensuring this - I am
> fixing it.
>
> I guess, there are two ways to fix it.
>
> a) Topology should be not initialized in the above code block
> (topologyInitialized = true) until application contexts are loaded from
> registry.
> b) Complete topology request event should be not be sent until application
> contexts are loaded from registry.
>
> Thanks,
> Raj.
>
> --
> Rajkumar Rajaratnam
> Committer & PMC Member, Apache Stratos
> Software Engineer, WSO2
>
> Mobile : +94777568639
> Blog : rajkumarr.com
>
--
Akila Ravihansa Perera
WSO2 Inc.; http://wso2.com/
Blog: http://ravihansa3000.blogspot.com
