In summary, we have decided to use the usage notification events emitted by heat (http://tinyurl.com/oultjm5). This should allow us to detect stack action completions, and errors (with reasons) which should be enough for a Bay implementation that does not need to poll. Stack timeouts can be passed to heat as parameters. This is not as elegant as what Angus and Zane suggested because it requires Magnum to access all RPC messages. With their proposed approach we could use a Zaqar queue that only emits the stack events relevant to that user/tenant. This conforms better to the principle of least privilege. Our preference for the decided approach is that it allows us tight integration with Heat using functionality that exists today. We admit that this implementation will be harder to debug than other options.
More remarks in-line. On Mar 11, 2015, at 2:27 AM, Jay Lau <jay.lau....@gmail.com> wrote: > > > 2015-03-10 23:21 GMT+08:00 Hongbin Lu <hongbin...@gmail.com>: > Hi Adrian, > > On Mon, Mar 9, 2015 at 6:53 PM, Adrian Otto <adrian.o...@rackspace.com> wrote: > Magnum Team, > > In the following review, we have the start of a discussion about how to > tackle bay status: > > https://review.openstack.org/159546 > > I think a key issue here is that we are not subscribing to an event feed from > Heat to tell us about each state transition, so we have a low degree of > confidence that our state will match the actual state of the stack in > real-time. At best, we have an eventually consistent state for Bay following > a bay creation. > > Here are some options for us to consider to solve this: > > 1) Propose enhancements to Heat (or learn about existing features) to emit a > set of notifications upon state changes to stack resources so the state can > be mirrored in the Bay resource. > > A drawback of this option is that it increases the difficulty of > trouble-shooting. In my experience of using Heat (SoftwareDeployments in > particular), Ironic and Trove, one of the most frequent errors I encountered > is that the provisioning resources stayed in deploying state (never went to > completed). The reason is that they were waiting a callback signal from the > provisioning resource to indicate its completion, but the callback signal was > blocked due to various reasons (e.g. incorrect firewall rules, incorrect > configs, etc.). Troubling-shooting such problem is generally harder. > I think that the "heat convergence" is working on the issues for your > concern: https://wiki.openstack.org/wiki/Heat/Blueprints/Convergence Yes, that wold address part of the concern, but not all of it. Depending on the implementation of state exchange, and the configuration of the network, it may be possible to create an installation of the system that does not function at all due to network ACLs, and almost no clue to the implementer about why it does not work. To mitigate this concern, we would want an implementation that does not require a bi-directional network trust relationship between Heat and Magnum. Instead, implement it in a way that connections are only made from Magnum to Heat, so if the stack creation call succeeds, so can the status updates. Perhaps we can revisit this design discussion in a future iteration of our Bay status code. Adrian > > 2) Spawn a task to poll the Heat stack resource for state changes, and > express them in the Bay status, and allow that task to exit once the stack > reaches its terminal (completed) state. > > 3) Don’t store any state in the Bay object, and simply query the heat stack > for status as needed. > > Are each of these options viable? Are there other options to consider? What > are the pro/con arguments for each? > > Thanks, > > Adrian > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > -- > Thanks, > > Jay Lau (Guangya Liu) > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
