By this I meant I'm fine with any decision the implementer makes, btw. We will encounter the administrative issues with it and deal with them.
On Tue, Dec 29, 2015 at 2:46 PM, Daan Hoogland <daan.hoogl...@gmail.com> wrote: > Wido, Rafael, > > I like the date-format but then of course CCYY-MM-DD. I can still think of > ways to screw up that (or the plain int;) > > On Tue, Dec 29, 2015 at 1:40 PM, Rafael Weingärtner < > rafaelweingart...@gmail.com> wrote: > >> Wido, that is true, you are right; the naming on upgrade routines can use >> a >> numeric value independent of the number of the version. The numeric value >> can be a simple integer that is incremented each routine that is added or >> a >> time stamp when the routine was added. The point is that we would have to >> link a version to a number. That would enable us to use flywaydb. >> >> To use that approach I think we might need to break compatibility as you >> pointed out earlier, but I believe that the benefits of an improved way to >> manage upgrade routines will compensate by the breaking of compatibility. >> >> On Tue, Dec 29, 2015 at 10:25 AM, Wido den Hollander <w...@widodh.nl> >> wrote: >> >> > >> > >> > On 29-12-15 13:21, Rafael Weingärtner wrote: >> > > I got your point Daan. >> > > >> > > Well, and if we linked a version of ACS with a time stamp in the >> format >> > of >> > > DD.MM.YYYY? >> > > >> > >> > In that case you could also say. >> > >> > ACS 4.6.0 == db ver X >> > >> > You don't have to say ver >= X, you can also say ver = X. >> > >> > > We could then use the time stamp in the same format to name upgrade >> > > routines. This way the idea of running all of the routines in between >> > > version during upgrades could be applied. >> > > >> > >> > Same goes for giving all database changes a simple numeric int which >> > keeps incrementing each time a change is applied ;) >> > >> > Wido >> > >> > > On Tue, Dec 29, 2015 at 10:03 AM, Daan Hoogland < >> daan.hoogl...@gmail.com >> > > >> > > wrote: >> > > >> > >> Rafael, >> > >> >> > >> On Tue, Dec 29, 2015 at 12:22 PM, Rafael Weingärtner < >> > >> rafaelweingart...@gmail.com> wrote: >> > >> >> > >>> Thanks, Daan and Wido for your contributions, I will discuss them as >> > >>> follows. >> > >>> >> > >>> Daan, about the idea of per commit upgrades. Do you mean that we >> > separate >> > >>> each change in the database that is introduced by PRs/Commits in a >> > >>> different file (routine upgrade) per ACS version? >> > >>> So we would have, V_480_A.sql (for a PR),V_480_B.sql (for another >> PR) >> > and >> > >>> so forth >> > >>> >> > >>> If that is the case, we can achieve that using a simple convention >> > naming >> > >>> as I suggested. Each developer when she/he needs to change or add >> > >> something >> > >>> in the database creates an upgrade routine separately and gives it >> an >> > >>> execution order to be taken by Flywaydb. I think that could help >> RMs to >> > >>> track and isolate the problem, right? >> > >>> >> > >> Yes, with one little caveat. We do not know in what version a >> > feature/PR >> > >> will end up at the time of implementing, so a name containing the >> > version >> > >> would not be ideal. >> > >> >> > >> >> > >> >> > >>> >> > >>> Hi Wido, now I understand your example. >> > >>> I understand your worry about upgrade paths, and that is the point I >> > want >> > >>> to discuss and solve. In your example, if we release a 4.6.0 and >> later >> > a >> > >>> 4.5.3. You said that there would be no upgrade path from 4.5.3 to >> > 4.6.0. >> > >>> Well, today that is what happens. However, if we change the >> technology >> > we >> > >>> use to upgrade the database (using a tool such as Flywaydb) and if >> we >> > >>> define a standard to create upgrade routines that would not be a >> > problem. >> > >>> >> > >>> As I have written in my first email, to go from a version to >> another we >> > >>> should be able to run all of the upgrade routines in between them >> > >>> (including the upgrade routine of the goal version). Therefore, if >> we >> > >>> release a version 4.6.0, and then 4.5.3, if someone upgrades to >> 4.5.3 >> > >> from >> > >>> any other version, and then wants to upgrade to 4.6.0, that would >> not >> > be >> > >> a >> > >>> problem, it would be a metter of running only the routine upgrade of >> > >> 4.6.0 >> > >>> version. We do not need to explicitly create upgrade paths. They >> should >> > >> be >> > >>> implicit by our upgrade conventions. >> > >>> >> > >>> About creating versions of the code that rely on some version of the >> > >>> database. I do not like much because of compatibility issues that >> might >> > >>> arise. For instance, let’s say version X of ACS depends on version >> >=Y >> > of >> > >>> the database. If I upgrade the database to version Y + 1 or +2, the >> > same >> > >>> ACS version has to keep running nice and shiny. My worry is that may >> > >> bring >> > >>> some complications, such as to remove columns that cease to be used >> or >> > >> data >> > >>> structure that we might want to improve. >> > >>> >> > >>> I normally see that the database version and the code base are tied >> in >> > a >> > >>> mapping 1 to 1. Maybe I am having troubles identifying the benefits >> of >> > >> that >> > >>> change. >> > >>> >> > >>> Thanks for your time ;) >> > >>> >> > >>> On Tue, Dec 29, 2015 at 8:15 AM, Wido den Hollander <w...@widodh.nl >> > >> > >>> wrote: >> > >>> >> > >>>> >> > >>>> >> > >>>> On 28-12-15 21:34, Rafael Weingärtner wrote: >> > >>>>> Hi Wido, Rohit, >> > >>>>> I have just read the feature suggestion. >> > >>>>> >> > >>>>> Wido, I am not trying to complicate things, quite the opposite, I >> > >> just >> > >>>>> illustrate a simple thing that can happen and is happening; I just >> > >>>> pointed >> > >>>>> how it can be easily solved. >> > >>>>> >> > >>>>> About the release of .Z, releases more constant and others, I do >> not >> > >>> want >> > >>>>> to mix topics. Let’s keep this thread strict to discuss database >> > >>>> upgrades. >> > >>>>> >> > >>>> >> > >>>> I do not want to start the release discussion, but what I meant is >> > that >> > >>>> we try to find a technical solution to something which might be >> solved >> > >>>> easier by just changing the way we release. >> > >>>> >> > >>>> 4.6.0 is released and afterwards 4.5.3 is released. How does >> somebody >> > >>>> upgrade from 4.5.3 to 4.6.0? He can't, since the 4.6.0 code doesn't >> > >>>> support that path. >> > >>>> >> > >>>> So my idea is to split the database version from the code version. >> > >>>> >> > >>>> The code requires database version >= X and during boot it simply >> > >> checks >> > >>>> that. >> > >>>> >> > >>>> The database migration tool can indeed do the DB migration, it >> doesn't >> > >>>> have to be the mgmt server who does the upgrade. >> > >>>> >> > >>>>> Now, about the FS. I agree with Rohit that we should have only one >> > >> way >> > >>> of >> > >>>>> managing database upgrades and creation. I just do not like the >> idea >> > >> of >> > >>>>> creating a tool that work as a wrapper on frameworks/tools such as >> > >>>>> flywaydb. I think that those frameworks already work pretty good >> as >> > >>> they >> > >>>>> are; and, I would rather maintain configurations than some wrapper >> > >>> code. >> > >>>>> >> > >>>>> I personally like the way ACS works during upgrades (I just do not >> > >> like >> > >>>> the >> > >>>>> code itself and how things are structured), as a system >> > >> administrator I >> > >>>>> like to change the version in the >> > >>>> “/etc/apt/sources.list.d/cloudstack.list” >> > >>>>> and use the "apt-get" "update" and "install" from the command >> line. I >> > >>> do >> > >>>>> not see the need to add another tool that is just a wrapper to the >> > >> mix. >> > >>>> If >> > >>>>> I update ACS code to 4.7.0, why would I let the database schema >> in an >> > >>>> older >> > >>>>> version? And if we want version DB schemas and application code >> > >>>> separately >> > >>>>> maintaining somehow compatibility between them, which would bring >> a >> > >>> whole >> > >>>>> other level of complexity to the code; I think we should avoid >> that. >> > >>>>> >> > >>>>> The flywaydb can be easily integrated with everything we have >> now; we >> > >>>> could >> > >>>>> have a maven profile for developers and integrate it in ACS >> bootstrap >> > >>>> using >> > >>>>> its API as a Spring bean. Therefore, we could remove the current >> > >>>>> “DatabaseUpgradeChecker “, “DbUpgrade” and other classes that aim >> to >> > >> do >> > >>>>> that. We could even add the creation of the schema into the first >> > >> time >> > >>> it >> > >>>>> boots using flywaydb and retire the “cloudstack-setup-database” >> > >> script, >> > >>>> or >> > >>>>> at least make it less complicated, using it just to configure the >> > >>>> database >> > >>>>> URL and users. >> > >>>>> >> > >>>>> The point is that to use Flywaydb we would have to agree upon a >> > >>>> convention >> > >>>>> on creating routines (java and SQL) to execute upgrades. Moreover, >> > >>> using >> > >>>> a >> > >>>>> tool such as Flywaydb we do not need to worry about upgrade paths. >> > >> As I >> > >>>>> wrote in the email I used to start this thread, the upgrade has >> to be >> > >>>>> straightforward, to go to a version we have to run all of the >> upgrade >> > >>>>> routines between the current version until the desired one. Our >> job >> > >> is >> > >>> to >> > >>>>> create upgrade routines that work and name them properly, the job >> of >> > >>> the >> > >>>>> tool is to check the current version, the desired one, the >> upgrades >> > >>> that >> > >>>> it >> > >>>>> needs to run and execute everything properly. >> > >>>>> >> > >>>> >> > >>>> Yes, indeed. I just wanted to start the discussion if we shouldn't >> > >>>> version the database differently from the code. >> > >>>> >> > >>>>> Additionally, I do not see the need to break compatibility as >> Rohit >> > >>>>> suggested in the FS; in my opinion, everything we have up today >> can >> > >> be >> > >>>>> migrated to the new structure I proposed. If we use a tool such as >> > >>>>> Flywaydb, I even volunteered for that. The only thing we have to >> > >>> discuss >> > >>>>> and agree upon is the naming conventions for upgrades routines, >> where >> > >>> to >> > >>>>> put them and the configurations for flywaydb. >> > >>>>> >> > >>>>> Thanks for your contribution and time. >> > >>>>> >> > >>>>> >> > >>>>> On Mon, Dec 28, 2015 at 2:10 PM, Rohit Yadav < >> > >>> rohit.ya...@shapeblue.com> >> > >>>>> wrote: >> > >>>>> >> > >>>>>> Hi Rafael and Wido, >> > >>>>>> >> > >>>>>> Thanks for starting a conversation in this regard, I could not >> > >> pursue >> > >>>> the >> > >>>>>> Chimp tool due to other $dayjob work though it’s good to see some >> > >>>>>> discussion has started again. Hope we’ll solve this in 2016. >> > >>>>>> >> > >>>>>> In my opinion, we will need to first separate the database >> > >>>> init/migration >> > >>>>>> tooling away from mgmt server (right now the mgmt server does db >> > >>>> migrations >> > >>>>>> when it starts and there is a code/db version mismatch) and >> secondly >> > >>>> make >> > >>>>>> sure that we’re using the same code/tool to deploy database >> (right >> > >>> now, >> > >>>>>> users use the cloudstack-setup-database python tool while >> developer >> > >>> use >> > >>>> the >> > >>>>>> maven/java DatabaseCreator activated by the -Ddeploydb flag). >> > >>>>>> >> > >>>>>> After we’ve addressed these two issues we can look into how we >> can >> > >>>> support >> > >>>>>> minor releases workflow (or decide to do something else, like not >> > >>>> support >> > >>>>>> .Z releases like Wido mentioned), and see if we can or want to >> use >> > >> any >> > >>>>>> existing migration tool or write a wrapper tool “chimp” that uses >> > >>>> existing >> > >>>>>> tools (some of those are mentioned in the Chimp FS like flywaydb >> > >> etc). >> > >>>> For >> > >>>>>> allowing users to go back and forth from a db schema/version, >> we’ll >> > >>> also >> > >>>>>> need some new DB migration >> > >>> conventions/versioning/rules/static-checking, >> > >>>>>> and how developer need to write such paths (forward and reverse) >> > >> etc. >> > >>>>>> >> > >>>>>> The best approach I figured at the time was to decide that we’ll >> use >> > >>> the >> > >>>>>> previous db upgrade path mechanism till a certain CloudStack >> version >> > >>>> (say >> > >>>>>> 4.8.0) and after that we’ll use the new approach or tooling to >> > >>>>>> upgrade/downgrade DB schemas (thereby retiring away from the old >> DB >> > >>>> upgrade >> > >>>>>> path mess). >> > >>>>>> >> > >>>>>>> >> > >>>>>> >> > >>>>>> [image: ShapeBlue] <http://www.shapeblue.com> Rohit Yadav >> Software >> > >>>>>> Architect , ShapeBlue d: * | s: +44 203 603 0540* >> > >>>>>> <%7C%20s:%20+44%20203%20603%200540> | m: *+91 8826230892* >> > >>>>>> <+91%208826230892> e: *rohit.ya...@shapeblue.com | t: * >> > >>>>>> <rohit.ya...@shapeblue.com%20%7C%20t:> | w: * >> www.shapeblue.com* >> > >>>>>> <http://www.shapeblue.com> a: >> > >>>>>> 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape Blue Ltd >> > >> is a >> > >>>>>> company incorporated in England & Wales. ShapeBlue Services India >> > >> LLP >> > >>>> is a >> > >>>>>> company incorporated in India and is operated under license from >> > >> Shape >> > >>>> Blue >> > >>>>>> Ltd. Shape Blue Brasil Consultoria Ltda is a company >> incorporated in >> > >>>> Brasil >> > >>>>>> and is operated under license from Shape Blue Ltd. ShapeBlue SA >> Pty >> > >>> Ltd >> > >>>> is >> > >>>>>> a company registered by The Republic of South Africa and is >> traded >> > >>> under >> > >>>>>> license from Shape Blue Ltd. ShapeBlue is a registered trademark. >> > >>>>>> This email and any attachments to it may be confidential and are >> > >>>> intended >> > >>>>>> solely for the use of the individual to whom it is addressed. Any >> > >>> views >> > >>>> or >> > >>>>>> opinions expressed are solely those of the author and do not >> > >>> necessarily >> > >>>>>> represent those of Shape Blue Ltd or related companies. If you >> are >> > >> not >> > >>>> the >> > >>>>>> intended recipient of this email, you must neither take any >> action >> > >>> based >> > >>>>>> upon its contents, nor copy or show it to anyone. Please contact >> the >> > >>>> sender >> > >>>>>> if you believe you have received this email in error. >> > >>>>>> >> > >>>>>> >> > >>>>>> On 28-Dec-2015, at 9:10 PM, Wido den Hollander <w...@widodh.nl> >> > >>> wrote: >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> On 28-12-15 16:21, Rafael Weingärtner wrote: >> > >>>>>>>> Thanks for your contribution Wido, >> > >>>>>>>> I have not seen Rohit’s email; I will take a look at it. >> > >>>>>>>> >> > >>>>>>> >> > >>>>>>> Ok, he has a FS here: >> > >>>>>>> >> > >>>> >> > >> >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+Chimp >> > >>>>>>> >> > >>>>>>>> About database schema changes happening only in X.Y, I also >> agree >> > >>> with >> > >>>>>> you >> > >>>>>>>> (that is a convention we all could agree on, and such as >> conding >> > >> and >> > >>>>>>>> release procedures we could have a wiki page for that). >> However, I >> > >>>>>> think we >> > >>>>>>>> still might have scripts in versions X.Y.Z to add data to a >> table >> > >>> such >> > >>>>>> as >> > >>>>>>>> “guest_os_hypervisor”. >> > >>>>>>>> >> > >>>>>>> >> > >>>>>>> Yes, that is true. A bugfix could be a addition into the >> database, >> > >>> but >> > >>>>>>> we have to prevent it as much as possible. >> > >>>>>>> >> > >>>>>>>> The point to manage such scripts is that, if we are in version >> > >> such >> > >>> as >> > >>>>>>>> 4.7.0 and a new script emerges in version 4.5.3, we would have >> to >> > >>>>>> decide to >> > >>>>>>>> run or not to run it. I would rather not run them, since if >> they >> > >> add >> > >>>>>>>> something to the code base; those changes should also be >> applied >> > >>> into >> > >>>>>>>> master and as a consequence it will be available in a future >> > >> update. >> > >>>>>>>> >> > >>>>>>> >> > >>>>>>> I understand, but this is where our release cycle becomes the >> > >>> problem. >> > >>>>>>> It is because we release a X.Y.Z release we run into these kind >> of >> > >>>>>> problems. >> > >>>>>>> >> > >>>>>>> If we as a project simple do not release the .Z releases we >> would >> > >> be >> > >>>>>>> fine as well ;) >> > >>>>>>> >> > >>>>>>> You can try to complicate things with technical things, or if we >> > >>>> release >> > >>>>>>> every two / three weeks we don't run into these kind of >> situations >> > >> :) >> > >>>>>>> >> > >>>>>>> We might even cut the database version loose from the code >> version. >> > >>>>>>> >> > >>>>>>> Database version is simple 100, 101, 102, 103, 104, 105. And a >> code >> > >>>>>>> version requires a certain version of the database. >> > >>>>>>> >> > >>>>>>> Wido >> > >>>>>>> >> > >>>>>>>> On Mon, Dec 28, 2015 at 12:50 PM, Wido den Hollander < >> > >>> w...@widodh.nl> >> > >>>>>> wrote: >> > >>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> On 28-12-15 14:16, Rafael Weingärtner wrote: >> > >>>>>>>>>> Hi all devs, >> > >>>>>>>>>> First of all, sorry the long text, but I hope we can start a >> > >>>>>> discussion >> > >>>>>>>>>> here and improve that part of ACS. >> > >>>>>>>>>> >> > >>>>>>>>>> A while ago I have faced the code that Apache CloudStack >> (ACS) >> > >>> uses >> > >>>> to >> > >>>>>>>>>> upgrade from a version to newer one and that did not seem to >> be >> > >> a >> > >>>> good >> > >>>>>>>>> way >> > >>>>>>>>>> to execute our upgrades. Therefore, I decided to use some >> time >> > >> to >> > >>>>>> search >> > >>>>>>>>>> for alternatives. >> > >>>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> I think we all saw that happen once or more :) >> > >>>>>>>>> >> > >>>>>>>>>> I have read some material about versioning of scripts used to >> > >>>> upgrade >> > >>>>>> a >> > >>>>>>>>>> database (DB) of a system and went through some frameworks >> that >> > >>>> could >> > >>>>>>>>> help >> > >>>>>>>>>> us. >> > >>>>>>>>>> >> > >>>>>>>>>> In the literature of software engineering, it is firmly >> stated >> > >>> that >> > >>>> we >> > >>>>>>>>> have >> > >>>>>>>>>> to version DB scripts as we do with the source code of the >> > >>>>>> application, >> > >>>>>>>>>> using the baseline approach. Gladly, we were not that bad at >> > >> this >> > >>>>>> point, >> > >>>>>>>>> we >> > >>>>>>>>>> already versioned our routines for DB upgrade (.sql and >> .java). >> > >>>>>>>>> Therefore, >> > >>>>>>>>>> it seemed that we just did not have used a practical >> approach to >> > >>>> help >> > >>>>>> us >> > >>>>>>>>>> during DB upgrades. >> > >>>>>>>>>> >> > >>>>>>>>>> From my readings and looking at the ACS source code I raised >> the >> > >>>>>>>>> following >> > >>>>>>>>>> requirement: >> > >>>>>>>>>> • We should be able to write more than one routine to upgrade >> > >> to a >> > >>>>>>>>>> version; those routines can be written in Java and SQL. We >> might >> > >>>> have >> > >>>>>>>>> more >> > >>>>>>>>>> than a routine to be executed for each version and we should >> be >> > >>> able >> > >>>>>> to >> > >>>>>>>>>> define an order of execution. Additionally, to go to an upper >> > >>>>>> version, we >> > >>>>>>>>>> have to run all of the routines from smaller versions first, >> > >> until >> > >>>> we >> > >>>>>>>>>> achieve the desired version. >> > >>>>>>>>>> >> > >>>>>>>>>> We could also add another requirement that is the downgrade >> > >> from a >> > >>>>>>>>> version, >> > >>>>>>>>>> which we currently do not support. With that comes my first >> > >>> question >> > >>>>>> for >> > >>>>>>>>>> discussion: >> > >>>>>>>>>> • Do we want/need a method to downgrade from a version to a >> > >>> previous >> > >>>>>>>>> one? >> > >>>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> I personally do not care. Usually people should create a >> backup >> > >>> PRIOR >> > >>>>>> to >> > >>>>>>>>> a upgrade. If that fails they can restore the backup. >> > >>>>>>>>> >> > >>>>>>>>>> I found an explanation for not supporting downgrades, and I >> > >> liked >> > >>>> it: >> > >>>>>>>>>> http://flywaydb.org/documentation/faq.html#downgrade >> > >>>>>>>>>> >> > >>>>>>>>>> So, what I devised for us: >> > >>>>>>>>>> First the bureaucracy part - our migrations occur basically >> in >> > >>> three >> > >>>>>> (3) >> > >>>>>>>>>> steps, first we have a "prepare script", then a cleanup >> script >> > >> and >> > >>>>>>>>> finally >> > >>>>>>>>>> the migration per se that is written in Java, at least, that >> is >> > >>> what >> > >>>>>> we >> > >>>>>>>>> can >> > >>>>>>>>>> expect when reading the interface >> > >>> “com.cloud.upgrade.dao.DbUpgrade”. >> > >>>>>>>>>> >> > >>>>>>>>>> Additionally, our scripts have the following naming >> convention: >> > >>>>>>>>>> schema-<currentVersion>to<desiredVersion>, which in IMHO may >> > >> cause >> > >>>>>> some >> > >>>>>>>>>> confusion because at first sight we may think that from the >> same >> > >>>>>> version >> > >>>>>>>>> we >> > >>>>>>>>>> could have different paths to an upper version, which in >> > >> practice >> > >>> is >> > >>>>>> not >> > >>>>>>>>>> happening. Instead of a <currentVersion>to<version> we could >> > >>> simply >> > >>>>>> use >> > >>>>>>>>>> V_<numberOfVersion>_<sequencial>.<fileExtension>, giving >> that, >> > >> we >> > >>>>>> have to >> > >>>>>>>>>> execute all of the V_<version> scripts that are smaller than >> the >> > >>>>>> version >> > >>>>>>>>> we >> > >>>>>>>>>> want to upgrade. >> > >>>>>>>>>> >> > >>>>>>>>>> To clarify what I am saying, I will use an example. Let’s >> say we >> > >>>> have >> > >>>>>>>>> just >> > >>>>>>>>>> installed ACS and ran the cloudstack-setup-database. That >> > >> command >> > >>>> will >> > >>>>>>>>>> create a database schema in version 4.0.0. To upgrade that >> > >> schema >> > >>> to >> > >>>>>>>>>> version 4.3.0 (it is just an example, it could be any other >> > >>>> version), >> > >>>>>> ACS >> > >>>>>>>>>> will use the following mapping: >> > >>>>>>>>>> >> > >>>>>>>>>> _upgradeMap.put("4.0.0", new DbUpgrade[] {new >> Upgrade40to41(), >> > >> new >> > >>>>>>>>>> Upgrade410to420(), new Upgrade420to421(), new >> Upgrade421to430()) >> > >>>>>>>>>> >> > >>>>>>>>>> After loading the mapping, ACS will execute the scripts >> defined >> > >> in >> > >>>>>> each >> > >>>>>>>>> one >> > >>>>>>>>>> of the Upgrade path classes and the migration code per se. >> > >>>>>>>>>> >> > >>>>>>>>>> Now, let’s say we change the “.sql” scripts name to the >> pattern >> > >> I >> > >>>>>>>>>> mentioned, we would have the following scripts; those are the >> > >>>> scripts >> > >>>>>>>>> found >> > >>>>>>>>>> that aim to upgrade to versions between the interval 4.0.0 – >> > >> 4.3.0 >> > >>>>>>>>>> (considering 4.3.0, since that is the goal version): >> > >>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>> - schema-40to410, can be named to: V_410_A.sql >> > >>>>>>>>>> - schema-40to410-cleanup, can be named to: V_410_B.sql >> > >>>>>>>>>> - schema-410to420, can be named to: V_420_A.sql >> > >>>>>>>>>> - schema-410to420-cleanup , can be named to: V_420_b.sql >> > >>>>>>>>>> - schema-420to421, can be named to: V_421_A.sql >> > >>>>>>>>>> - schema-421to430, can be named to: V_430_A.sql >> > >>>>>>>>>> - schema-421to430-cleanup, can be named to: V_430_B.sql >> > >>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>> Additionally, all of the java code would have to follow the >> same >> > >>>>>>>>>> convention. For instance, we have >> > >>>>>> “com.cloud.upgrade.dao.Upgrade40to41”, >> > >>>>>>>>>> which has some java code to migrate from 4.0.0 to 4.1.0. The >> > >> idea >> > >>> is >> > >>>>>> to >> > >>>>>>>>>> extract that migration code to a Java class named: >> V_410_C.java, >> > >>>>>> giving >> > >>>>>>>>>> that it has to execute the SQL scripts before the java code. >> > >>>>>>>>>> >> > >>>>>>>>>> In order to go from a smaller version (4.0.0) to an upper one >> > >>>>>> (4.3.0), we >> > >>>>>>>>>> have to run all of the migration routines from intermediate >> > >>>> versions. >> > >>>>>>>>> That >> > >>>>>>>>>> is what we are already doing, but we do all of that manually. >> > >>>>>>>>>> >> > >>>>>>>>>> Bottom line, I think we could simple use the convention >> > >>>>>>>>>> V_<numberOfVersion>_<sequencial>.<fileExtension> to name >> upgrade >> > >>>>>>>>> routines. >> > >>>>>>>>>> That would facilitate us to use a framework to help us with >> that >> > >>>>>> process. >> > >>>>>>>>>> Additionally, I believe that we should always assume that to >> go >> > >>>> from a >> > >>>>>>>>>> smaller version to a higher one, we should run all of the >> > >> scripts >> > >>>> that >> > >>>>>>>>>> exist between them. What do you guys think of that? >> > >>>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> That seems good to me. But we still have to prevent that we >> > >> perform >> > >>>>>>>>> database changes in a X.Y.Z release since that is branched off >> > >> to a >> > >>>>>>>>> different branch. >> > >>>>>>>>> >> > >>>>>>>>> Imho database changes should only happen in X.Y releases. >> > >>>>>>>>> >> > >>>>>>>>>> After the bureaucracy, we can discuss tools. If we use that >> > >>>>>> convention to >> > >>>>>>>>>> name migration (upgrade) routines, we can start thinking on >> > >> tools >> > >>> to >> > >>>>>>>>>> support our migration process. I found two (2) promising >> ones: >> > >>>>>> Liquibase >> > >>>>>>>>>> and Flywaydb (both seem to be under Apache license, but the >> > >> first >> > >>>> one >> > >>>>>> has >> > >>>>>>>>>> an enterprise version?!). After reading the documentation and >> > >> some >> > >>>>>> usage >> > >>>>>>>>>> examples I found the flywaydb easier and simpler to use. >> > >>>>>>>>>> >> > >>>>>>>>>> What are the options of tools that we can use to help us >> manage >> > >>> the >> > >>>>>>>>>> database upgrade, without needing to code the upgrade path >> that >> > >>> you >> > >>>>>> know? >> > >>>>>>>>>> >> > >>>>>>>>>> After that, I think we should decide if we should create >> another >> > >>>>>>>>>> project/component to take care of migrations, or we can just >> add >> > >>> the >> > >>>>>>>>>> dependency of the tool to a project such as >> “cloud-framework-db” >> > >>> and >> > >>>>>>>>> start >> > >>>>>>>>>> using it. >> > >>>>>>>>>> >> > >>>>>>>>>> The “cloud-framework-db” project seems to have a focus on >> other >> > >>>> things >> > >>>>>>>>> such >> > >>>>>>>>>> as managing transactions and generating SQLs from annotations >> > >> (?!? >> > >>>>>> That >> > >>>>>>>>>> should be a topic for another discussion). Therefore, I would >> > >>> rather >> > >>>>>>>>> create >> > >>>>>>>>>> a new project that has the specific goal of managing ACS DB >> > >>>> upgrades. >> > >>>>>> I >> > >>>>>>>>>> would also move all of the routines (SQL and Java) to this >> new >> > >>>>>> project. >> > >>>>>>>>>> This project would be a module of the CloudStack project and >> it >> > >>>> would >> > >>>>>>>>>> execute the upgrade routines at the startup of ACS. >> > >>>>>>>>>> >> > >>>>>>>>>> I believe that going from a homemade solution to one that is >> > >> more >> > >>>>>>>>>> consolidated and used by other communities would be the way >> to >> > >> go. >> > >>>>>>>>>> >> > >>>>>>>>>> I can volunteer myself to create a PR with the aforementioned >> > >>>> changes >> > >>>>>> and >> > >>>>>>>>>> using flywaydb to manage our upgrades. However, I prefer to >> > >> have a >> > >>>>>> good >> > >>>>>>>>>> discussion with other devs first, before starting coding. >> > >>>>>>>>>> >> > >>>>>>>>>> Do you have suggestions or points that should be raised >> before >> > >> we >> > >>>>>> start >> > >>>>>>>>>> working on that? >> > >>>>>>>>> >> > >>>>>>>>> Rohit suggested Chimp earlier this year: >> > >>>>>>>>> >> > >>>>>>>>> >> > >>>>>> >> > >>>> >> > >>> >> > >> >> > >> http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201508.mbox/%3c677bd09f-fc75-4888-8dc8-2b7af7439...@shapeblue.com%3E >> > >>>>>>>>> >> > >>>>>>>>> The thread is called: "[DISCUSS] Let's fix CloudStack Upgrades >> > >> and >> > >>> DB >> > >>>>>>>>> migrations with CloudStack Chimp" >> > >>>>>>>>> >> > >>>>>>>>> Maybe there is something good in there. >> > >>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>> -- >> > >>>>>>>>>> Rafael Weingärtner >> > >>>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>> >> > >>>>>> Regards. >> > >>>>>> >> > >>>>>> Find out more about ShapeBlue and our range of CloudStack related >> > >>>> services: >> > >>>>>> IaaS Cloud Design & Build >> > >>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – >> > >>> rapid >> > >>>>>> IaaS deployment framework <http://shapeblue.com/csforge/> >> > >>>>>> CloudStack Consulting < >> http://shapeblue.com/cloudstack-consultancy/ >> > >>> >> > >>> | >> > >>>> CloudStack >> > >>>>>> Software Engineering >> > >>>>>> <http://shapeblue.com/cloudstack-software-engineering/> >> > >>>>>> CloudStack Infrastructure Support >> > >>>>>> <http://shapeblue.com/cloudstack-infrastructure-support/> | >> > >>> CloudStack >> > >>>>>> Bootcamp Training Courses < >> > >> http://shapeblue.com/cloudstack-training/> >> > >>>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>> >> > >>> >> > >>> >> > >>> >> > >>> -- >> > >>> Rafael Weingärtner >> > >>> >> > >> >> > >> >> > >> >> > >> -- >> > >> Daan >> > >> >> > > >> > > >> > > >> > >> >> >> >> -- >> Rafael Weingärtner >> > > > > -- > Daan > -- Daan
