Hi folks, Recently in the Inria's Discovery initiative[1], we got in touch with CockroachLabs guys with an idea: make Keystone supports CockorachDB. So we give it a try and you can find a very first result on our GitHub[2]. The GitHub project consists of a Vagrant file that spawns a VM with a CockroachDB database and then installs Keystone with Devstack using CockroachDB as backend.
CockroachDB claims to support the PostgreSQL protocol. It also provides support for SQLAlchemy that mostly inherits from the PostgreSQL one. So, making Keystone working with CockroachDB should be easy peasy right? Almost! The rest of this email describes what we have done, to make it works. sqlalchemy-migrate doesn't work with CockroachDB ================================================ Keystone uses a database migration tool named sqlalchemy-migrate[3]. This tool is called during the deployment of Keystone to migrate the database from its initial versions to its actual version. Unfortunately, migration failed with the following (partial) stacktrace: ,---- | DEBUG migrate.versioning.repository [-] Config: OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'), ('repository_id', 'keystone'), ('version_table', 'migrate_version'), ('required_dbs', '[]'), ('use_timestamp_numbering', 'False')]))]) __init__ /usr/local/lib/python2.7/dist-packages/migrate/versioning/repository.py:83 | INFO migrate.versioning.api [-] 66 -> 67... | CRITICAL keystone [-] KeyError: 'cockroachdb' | ... | TRACE keystone visitorcallable = get_engine_visitor(engine, visitor_name) | TRACE keystone File "/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py", line 47, in get_engine_visitor | TRACE keystone return get_dialect_visitor(engine.dialect, name) | TRACE keystone File "/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py", line 62, in get_dialect_visitor | TRACE keystone migrate_dialect_cls = DIALECTS[sa_dialect_name] | TRACE keystone KeyError: 'cockroachdb' `---- As we understand it, sqlalchemy-migrate contains dedicated SQL backend codes and there is no such code for CockroachDB. As a workaround, we have performed a second OS deployment with PostgreSQL as backend and made a dump of the database after migrations. We then bypassed migrations in our first deployment by setting up CockroachDB with the dumped script. Note that we had to edit the pgdump script a little bit because some features are missing in CockroachDB. The workaround we have to go with represents an obstacle to test CockroachDB with other OpenStack core services. We would be grateful if some of you would help us with adding the support of CockroachDB in sqlalchemy-migrate. oslo.db contains backend specific code for error handling ========================================================= The `oslo.db' library contains one file with backend-specific codes (`oslo_db/sqlalchemy/exc_filters.py'[4]). This file aims at abstracting database exceptions that differ but target the same class of error (because each backend produces a specific exception with a specific error message). Intercepted exceptions are wrapped into a common OS exception to abstract backend errors in the rest of `oslo.db'. CockroachDB doesn't produce same errors than PostgreSQL, so we have to update this class. Note that our POC is not exhaustive since we only added error messages we saw during Tempest/Rally tests. You can look at the differences between OpenStack/oslo.db/stable/pike and our fork on GitHub[5]. We only add two lines! CockroachDB manages isolation without lock ========================================== CockroachDB lets you write transactions that respect ACID properties. Regarding the "I" (Isolation), CockroachDB offers snapshot and serializable isolation but doesn't rely on locks to do that. So every time there is concurrent editing transactions that end in a conflict, then you have to retry the transactions. Fortunately, `oslo.db' already offers a decorator[6] to do that automatically. But, based on tests we run with Tempest/Rally, we figured out that some Keystone's SQL requests needed the decorator. You can look at the differences between OpenStack/keystone/stable/pike and our fork on GitHub[7]. What's next? ============ You can drop yourself on the VM as a stack user and run Rally tests (see README). Tempest is disabled because we have an issue with its installation[8]. Note that modifications we made to make it works are minimal. This is promising for the adoption of CockroachDB by other core services. Nonetheless: - Fixing the problem with sqlalchemy-migrate will be helpful to ease the deployment process. If you are willing to help, we will be really grateful. - We have to look through the performance prism: Keystone over CockroachDB vs. Keystone over Galera. This part may involve more modifications of `oslo.db'. One thing we have in mind is the management of retrying transactions since CockroachDB's documentation suggests an approach[9] that doesn't match with actual `oslo.db' implementation. Best, [1] https://beyondtheclouds.github.io/ [2] https://github.com/BeyondTheClouds/openstack-cockroachdb-dev/ [3] https://github.com/openstack/sqlalchemy-migrate/tree/master/ [4] https://github.com/openstack/oslo.db/blob/stable/pike/oslo_db/sqlalchemy/exc_filters.py [5] https://github.com/openstack/oslo.db/compare/openstack:stable/pike...BeyondTheClouds:cockroachdb/pike [6] https://github.com/openstack/oslo.db/blob/stable/pike/oslo_db/api.py#L84 [7] https://github.com/openstack/keystone/compare/stable/pike...BeyondTheClouds:cockroachdb/pike [8] https://bugs.launchpad.net/devstack/+bug/1714981 [9] https://www.cockroachlabs.com/docs/stable/transactions.html#transaction-retries -- Ronan-A. Cherrueau https://rcherrueau.github.io __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev