On 12/23/2015 01:01 PM, Steven Hardy wrote:
On Tue, Dec 22, 2015 at 03:36:02PM +0000, Dougal Matthews wrote:Hi all,This topic came up in the 2015-12-15 meeting[1], and again briefly today. After working with the code that came out of the deployment library spec[2] I had some concerns with how we are storing the templates. Simply put, when we are dealing with 100+ files from tripleo-heat-templates how can we ensure consistency in Swift without any atomicity or transactions. I think this is best explained with a couple of examples.  - When we create a new deployment plan (upload all the templates to swift)   how do we handle the case where there is an error? For example, if we are   uploading 10 files - what do we do if the 5th one fails for some reason?   There is a patch to do a manual rollback[3], but I have concerns about   doing this in Python. If Swift is completely inaccessible for a short   period the rollback wont work either.How does using a different data store help fix this error path? Regardless of the Swift/DB/Git choice, you have to detect something failed, and mark the plan as in a failed state. - When deploying to Heat, we need to download all the YAML files from Swift.   This can take a couple of seconds. What happens if somebody starts to   upload a new version of the plan in the middle? We could end up trying to   deploy half old and half new files. We wouldn't have a consistent view of   the database.If this happens, the API design is wrong - we should have a plan reference a unique identifier, including a version (e.g through swift versioned objects, git tags/branches or whatever), then on deployment heat should download exactly one version of those artefacts (e.g via a swift tempurl or a git URL, or whatever). FYI heatclient already supports downloading template objects directly from swift, and heat/heatclient already support downloading from http URLs, so all we have to do AFAICS is generate a URL pointing to a consistent version of the templates.We had a few suggestions in the meeting:  - Add a locking mechanism. I would be concerned about deadlocks or having to   lock for the full duration of a deploy.I think we need to create a new version of a plan every time it's modified, then on deployment the concern around locking is much reduced, or even eliminated? - Store the files in a tarball (so we only deal with one file). I think we   could still hit issues with multiple operations unless we guarantee that   only one instance of the API is ever running. I think these could potentially be made to work, but at this point are we using the best tool for the job? For alternatives, I see a can think of a couple of options: - Use a database, like we did for Tuskar and most OpenStack API's do.Storing template files in a database with Tuskar was a mistake - we should *not* do that again IMO - it was a very opaque interface and caused a lot of confusion in use. Storing code (in this case yaml files) in some sort of versioned repository is *such* a solved problem - it'll be total wheel-reinventing if we implement template revision control inside a bespoke DB IMHO. I think swift is probably a workable solution (using versioning and tempurl's), but we should really consider making the template store pluggable, as having the API backed by an (operator visible) git repo is likely to be a nicer and more familiar solution. The question around validation data I'm less clear on - we already store discovery data in swift, which seems to work OK - how is the validation data different, in that it warrants a bespoke DB data store?
Each validation result is quite small (basically passed/failed, date&time of the run, and which stage and validation it corresponds to).
We'll want to be able to query them per stage, validation and plan, sort them by the time and possibly use select validations within a certain time span.
We can just store them in Swift, but then we'd have to write the code that maintains the indexes and filters the objects based on a given query. The validations can run in parallel, so we need to make sure the indexes are updated atomically (or not use indexes at all and always load all the results).
All that is certainly feasible, but it's also something that a database does out of the box. We're talking about a single table and 3-4 SELECT statements (or the equivalent in Mongo, etc.).
Steve __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
