Hi Guys, from the user's perspective, there are two points which come up again and again -
1. lack a prescribed roll back if an upgrade goes badly 2. The downtime involved in doing upgrades. - Upgrades are seen as CloudStack's biggest 'issue'. I've had to rescue enough upgrades to understand how complicated it is; however with the increased release velocity, the admin's experience of doing these upgrades needs to be taken into account or we will lose users because of the increased admin overhead and downtime. The purpose of Rohit's CloudChimp was to find a suitable tool/method to carry out schema changes *without downtime*. You guys are far better placed to argue the merits of any one solution than me. I would just ask that you keep in mind what the users are looking for - relatively clean and recoverable upgrade process. [ShapeBlue]<http://www.shapeblue.com> Paul Angus VP Technology , ShapeBlue d: +44 203 617 0528 | s: +44 203 603 0540<tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540> | m: +44 7711 418784<tel:+44%207711%20418784> e: paul.an...@shapeblue.com | t: @cloudyangus<mailto:paul.an...@shapeblue.com%20|%20t:%20@cloudyangus> | w: www.shapeblue.com<http://www.shapeblue.com> a: 53 Chandos Place, Covent Garden London WC2N 4HS UK [cid:image182380.png@8ca21c21.40847519] Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark. This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. -----Original Message----- From: Erik Weber [mailto:terbol...@gmail.com] Sent: 29 December 2015 21:45 To: dev <dev@cloudstack.apache.org> Subject: Re: Let’s discuss database upgrades On Mon, Dec 28, 2015 at 2:16 PM, Rafael Weingärtner < rafaelweingart...@gmail.com> wrote: > Hi all devs, > First of all, sorry the long text, but I hope we can start a > discussion here and improve that part of ACS. > > A while ago I have faced the code that Apache CloudStack (ACS) uses to > upgrade from a version to newer one and that did not seem to be a good > way to execute our upgrades. Therefore, I decided to use some time to > search for alternatives. > > I have read some material about versioning of scripts used to upgrade > a database (DB) of a system and went through some frameworks that > could help us. > > In the literature of software engineering, it is firmly stated that we > have to version DB scripts as we do with the source code of the > application, using the baseline approach. Gladly, we were not that bad > at this point, we already versioned our routines for DB upgrade (.sql > and .java). Therefore, it seemed that we just did not have used a > practical approach to help us during DB upgrades. > > From my readings and looking at the ACS source code I raised the > following > requirement: > • We should be able to write more than one routine to upgrade to a > version; those routines can be written in Java and SQL. We might have > more than a routine to be executed for each version and we should be > able to define an order of execution. Additionally, to go to an upper > version, we have to run all of the routines from smaller versions > first, until we achieve the desired version. > > We could also add another requirement that is the downgrade from a > version, which we currently do not support. With that comes my first > question for > discussion: > • Do we want/need a method to downgrade from a version to a previous > one? > > I found an explanation for not supporting downgrades, and I liked it: > http://flywaydb.org/documentation/faq.html#downgrade > > So, what I devised for us: > First the bureaucracy part - our migrations occur basically in three > (3) steps, first we have a "prepare script", then a cleanup script and > finally the migration per se that is written in Java, at least, that > is what we can expect when reading the interface > “com.cloud.upgrade.dao.DbUpgrade”. > > Additionally, our scripts have the following naming convention: > schema-<currentVersion>to<desiredVersion>, which in IMHO may cause > some confusion because at first sight we may think that from the same > version we could have different paths to an upper version, which in > practice is not happening. Instead of a <currentVersion>to<version> we > could simply use V_<numberOfVersion>_<sequencial>.<fileExtension>, > giving that, we have to execute all of the V_<version> scripts that > are smaller than the version we want to upgrade. > > To clarify what I am saying, I will use an example. Let’s say we have > just installed ACS and ran the cloudstack-setup-database. That command > will create a database schema in version 4.0.0. To upgrade that schema > to version 4.3.0 (it is just an example, it could be any other > version), ACS will use the following mapping: > > _upgradeMap.put("4.0.0", new DbUpgrade[] {new Upgrade40to41(), new > Upgrade410to420(), new Upgrade420to421(), new Upgrade421to430()) > > After loading the mapping, ACS will execute the scripts defined in > each one of the Upgrade path classes and the migration code per se. > > Now, let’s say we change the “.sql” scripts name to the pattern I > mentioned, we would have the following scripts; those are the scripts > found that aim to upgrade to versions between the interval 4.0.0 – > 4.3.0 (considering 4.3.0, since that is the goal version): > > > - schema-40to410, can be named to: V_410_A.sql > - schema-40to410-cleanup, can be named to: V_410_B.sql > - schema-410to420, can be named to: V_420_A.sql > - schema-410to420-cleanup , can be named to: V_420_b.sql > - schema-420to421, can be named to: V_421_A.sql > - schema-421to430, can be named to: V_430_A.sql > - schema-421to430-cleanup, can be named to: V_430_B.sql > > > Additionally, all of the java code would have to follow the same > convention. For instance, we have > “com.cloud.upgrade.dao.Upgrade40to41”, > which has some java code to migrate from 4.0.0 to 4.1.0. The idea is > to extract that migration code to a Java class named: V_410_C.java, > giving that it has to execute the SQL scripts before the java code. > > In order to go from a smaller version (4.0.0) to an upper one (4.3.0), > we have to run all of the migration routines from intermediate > versions. That is what we are already doing, but we do all of that manually. > > Bottom line, I think we could simple use the convention > V_<numberOfVersion>_<sequencial>.<fileExtension> to name upgrade routines. > That would facilitate us to use a framework to help us with that process. > Additionally, I believe that we should always assume that to go from a > smaller version to a higher one, we should run all of the scripts that > exist between them. What do you guys think of that? > > After the bureaucracy, we can discuss tools. If we use that convention > to name migration (upgrade) routines, we can start thinking on tools > to support our migration process. I found two (2) promising ones: > Liquibase and Flywaydb (both seem to be under Apache license, but the > first one has an enterprise version?!). After reading the > documentation and some usage examples I found the flywaydb easier and simpler > to use. > > What are the options of tools that we can use to help us manage the > database upgrade, without needing to code the upgrade path that you know? > > After that, I think we should decide if we should create another > project/component to take care of migrations, or we can just add the > dependency of the tool to a project such as “cloud-framework-db” and > start using it. > > The “cloud-framework-db” project seems to have a focus on other things > such as managing transactions and generating SQLs from annotations > (?!? That should be a topic for another discussion). Therefore, I > would rather create a new project that has the specific goal of > managing ACS DB upgrades. I would also move all of the routines (SQL and > Java) to this new project. > This project would be a module of the CloudStack project and it would > execute the upgrade routines at the startup of ACS. > > I believe that going from a homemade solution to one that is more > consolidated and used by other communities would be the way to go. > > I can volunteer myself to create a PR with the aforementioned changes > and using flywaydb to manage our upgrades. However, I prefer to have a > good discussion with other devs first, before starting coding. > > Do you have suggestions or points that should be raised before we > start working on that? > This isn't my field of work, so forgive me if this is self explanatory or something, but is there no tool like terraform/puppet or similar for database work? I mean, where you state you desired state and the tool handles it. To me it sounds like a good way would be if you could specify what you want to exist (or not), and how it should look like. "I want table XYZ to exist with THESE columns having THIS type(s) and THIS default value bla bla bla" Rather than handling a bunch of sql scripts that has to handle different mysql versions (come to think about an issue with a mariadb version crashing recently), a variety of cloudstack versions and a whole lot more. Disclaimer: i have no idea if this is what flywaydb does, if it is, then just ignore this. -- Erik Find out more about ShapeBlue and our range of CloudStack related services: IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
