CCYY == YYYY On Tue, Dec 29, 2015 at 3:06 PM, Rafael Weingärtner < rafaelweingart...@gmail.com> wrote:
> I also liked the date-format, what did you mean with CCYY? > > > > The way I think we might have a problem, would be to commits/PRs that end > up creating files with same names. Then, we would have to agree upon a way > to solve those conflicts, such as appending an extra character to indicate > a sequence to be followed or adding more data such as HH and mm to the > naming convention (YYYY-MM-DD-HH-mm). > > > I liked the way Wido suggested, we could just remove the “-” from > “YYYY-MM-DD-HH-mm” and use the value as an integer (YYYYMMDDHHmm). > > > > It seems that we are reaching a consensus. I would love to hear back from > other devs though, especially committers. > > > > BTW: do I have permission to create a page on the wiki so I can add > everything we discuss and agree upon here? This way, we could add that page > to the guidelines for devs creating PRs and committers reviewing and > merging them. > > On Tue, Dec 29, 2015 at 12:00 PM, Wido den Hollander <w...@widodh.nl> > wrote: > > > > > > > On 29-12-15 14:46, Daan Hoogland wrote: > > > Wido, Rafael, > > > > > > I like the date-format but then of course CCYY-MM-DD. I can still think > > of > > > ways to screw up that (or the plain int;) > > > > > > > 20151229 is a valid integer which you can simply use to compare with. > > > > 100, 101, 102 or 20151229, 20160103, 20160104, I don't care that much. > > > > My point is that the database version should be separated from the code > > base. > > > > Wido > > > > > On Tue, Dec 29, 2015 at 1:40 PM, Rafael Weingärtner < > > > rafaelweingart...@gmail.com> wrote: > > > > > >> Wido, that is true, you are right; the naming on upgrade routines can > > use a > > >> numeric value independent of the number of the version. The numeric > > value > > >> can be a simple integer that is incremented each routine that is added > > or a > > >> time stamp when the routine was added. The point is that we would have > > to > > >> link a version to a number. That would enable us to use flywaydb. > > >> > > >> To use that approach I think we might need to break compatibility as > you > > >> pointed out earlier, but I believe that the benefits of an improved > way > > to > > >> manage upgrade routines will compensate by the breaking of > > compatibility. > > >> > > >> On Tue, Dec 29, 2015 at 10:25 AM, Wido den Hollander <w...@widodh.nl> > > >> wrote: > > >> > > >>> > > >>> > > >>> On 29-12-15 13:21, Rafael Weingärtner wrote: > > >>>> I got your point Daan. > > >>>> > > >>>> Well, and if we linked a version of ACS with a time stamp in the > > format > > >>> of > > >>>> DD.MM.YYYY? > > >>>> > > >>> > > >>> In that case you could also say. > > >>> > > >>> ACS 4.6.0 == db ver X > > >>> > > >>> You don't have to say ver >= X, you can also say ver = X. > > >>> > > >>>> We could then use the time stamp in the same format to name upgrade > > >>>> routines. This way the idea of running all of the routines in > between > > >>>> version during upgrades could be applied. > > >>>> > > >>> > > >>> Same goes for giving all database changes a simple numeric int which > > >>> keeps incrementing each time a change is applied ;) > > >>> > > >>> Wido > > >>> > > >>>> On Tue, Dec 29, 2015 at 10:03 AM, Daan Hoogland < > > >> daan.hoogl...@gmail.com > > >>>> > > >>>> wrote: > > >>>> > > >>>>> Rafael, > > >>>>> > > >>>>> On Tue, Dec 29, 2015 at 12:22 PM, Rafael Weingärtner < > > >>>>> rafaelweingart...@gmail.com> wrote: > > >>>>> > > >>>>>> Thanks, Daan and Wido for your contributions, I will discuss them > as > > >>>>>> follows. > > >>>>>> > > >>>>>> Daan, about the idea of per commit upgrades. Do you mean that we > > >>> separate > > >>>>>> each change in the database that is introduced by PRs/Commits in a > > >>>>>> different file (routine upgrade) per ACS version? > > >>>>>> So we would have, V_480_A.sql (for a PR),V_480_B.sql (for another > > PR) > > >>> and > > >>>>>> so forth > > >>>>>> > > >>>>>> If that is the case, we can achieve that using a simple convention > > >>> naming > > >>>>>> as I suggested. Each developer when she/he needs to change or add > > >>>>> something > > >>>>>> in the database creates an upgrade routine separately and gives it > > an > > >>>>>> execution order to be taken by Flywaydb. I think that could help > RMs > > >> to > > >>>>>> track and isolate the problem, right? > > >>>>>> > > >>>>> Yes, with one little caveat. We do not know in what version a > > >>> feature/PR > > >>>>> will end up at the time of implementing, so a name containing the > > >>> version > > >>>>> would not be ideal. > > >>>>> > > >>>>> > > >>>>> > > >>>>>> > > >>>>>> Hi Wido, now I understand your example. > > >>>>>> I understand your worry about upgrade paths, and that is the > point I > > >>> want > > >>>>>> to discuss and solve. In your example, if we release a 4.6.0 and > > >> later > > >>> a > > >>>>>> 4.5.3. You said that there would be no upgrade path from 4.5.3 to > > >>> 4.6.0. > > >>>>>> Well, today that is what happens. However, if we change the > > >> technology > > >>> we > > >>>>>> use to upgrade the database (using a tool such as Flywaydb) and if > > we > > >>>>>> define a standard to create upgrade routines that would not be a > > >>> problem. > > >>>>>> > > >>>>>> As I have written in my first email, to go from a version to > another > > >> we > > >>>>>> should be able to run all of the upgrade routines in between them > > >>>>>> (including the upgrade routine of the goal version). Therefore, if > > we > > >>>>>> release a version 4.6.0, and then 4.5.3, if someone upgrades to > > 4.5.3 > > >>>>> from > > >>>>>> any other version, and then wants to upgrade to 4.6.0, that would > > not > > >>> be > > >>>>> a > > >>>>>> problem, it would be a metter of running only the routine upgrade > of > > >>>>> 4.6.0 > > >>>>>> version. We do not need to explicitly create upgrade paths. They > > >> should > > >>>>> be > > >>>>>> implicit by our upgrade conventions. > > >>>>>> > > >>>>>> About creating versions of the code that rely on some version of > the > > >>>>>> database. I do not like much because of compatibility issues that > > >> might > > >>>>>> arise. For instance, let’s say version X of ACS depends on version > > >>> =Y > > >>> of > > >>>>>> the database. If I upgrade the database to version Y + 1 or +2, > the > > >>> same > > >>>>>> ACS version has to keep running nice and shiny. My worry is that > may > > >>>>> bring > > >>>>>> some complications, such as to remove columns that cease to be > used > > >> or > > >>>>> data > > >>>>>> structure that we might want to improve. > > >>>>>> > > >>>>>> I normally see that the database version and the code base are > tied > > >> in > > >>> a > > >>>>>> mapping 1 to 1. Maybe I am having troubles identifying the > benefits > > >> of > > >>>>> that > > >>>>>> change. > > >>>>>> > > >>>>>> Thanks for your time ;) > > >>>>>> > > >>>>>> On Tue, Dec 29, 2015 at 8:15 AM, Wido den Hollander < > w...@widodh.nl > > > > > >>>>>> wrote: > > >>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> On 28-12-15 21:34, Rafael Weingärtner wrote: > > >>>>>>>> Hi Wido, Rohit, > > >>>>>>>> I have just read the feature suggestion. > > >>>>>>>> > > >>>>>>>> Wido, I am not trying to complicate things, quite the opposite, > I > > >>>>> just > > >>>>>>>> illustrate a simple thing that can happen and is happening; I > just > > >>>>>>> pointed > > >>>>>>>> how it can be easily solved. > > >>>>>>>> > > >>>>>>>> About the release of .Z, releases more constant and others, I do > > >> not > > >>>>>> want > > >>>>>>>> to mix topics. Let’s keep this thread strict to discuss database > > >>>>>>> upgrades. > > >>>>>>>> > > >>>>>>> > > >>>>>>> I do not want to start the release discussion, but what I meant > is > > >>> that > > >>>>>>> we try to find a technical solution to something which might be > > >> solved > > >>>>>>> easier by just changing the way we release. > > >>>>>>> > > >>>>>>> 4.6.0 is released and afterwards 4.5.3 is released. How does > > >> somebody > > >>>>>>> upgrade from 4.5.3 to 4.6.0? He can't, since the 4.6.0 code > doesn't > > >>>>>>> support that path. > > >>>>>>> > > >>>>>>> So my idea is to split the database version from the code > version. > > >>>>>>> > > >>>>>>> The code requires database version >= X and during boot it simply > > >>>>> checks > > >>>>>>> that. > > >>>>>>> > > >>>>>>> The database migration tool can indeed do the DB migration, it > > >> doesn't > > >>>>>>> have to be the mgmt server who does the upgrade. > > >>>>>>> > > >>>>>>>> Now, about the FS. I agree with Rohit that we should have only > one > > >>>>> way > > >>>>>> of > > >>>>>>>> managing database upgrades and creation. I just do not like the > > >> idea > > >>>>> of > > >>>>>>>> creating a tool that work as a wrapper on frameworks/tools such > as > > >>>>>>>> flywaydb. I think that those frameworks already work pretty good > > as > > >>>>>> they > > >>>>>>>> are; and, I would rather maintain configurations than some > wrapper > > >>>>>> code. > > >>>>>>>> > > >>>>>>>> I personally like the way ACS works during upgrades (I just do > not > > >>>>> like > > >>>>>>> the > > >>>>>>>> code itself and how things are structured), as a system > > >>>>> administrator I > > >>>>>>>> like to change the version in the > > >>>>>>> “/etc/apt/sources.list.d/cloudstack.list” > > >>>>>>>> and use the "apt-get" "update" and "install" from the command > > >> line. I > > >>>>>> do > > >>>>>>>> not see the need to add another tool that is just a wrapper to > the > > >>>>> mix. > > >>>>>>> If > > >>>>>>>> I update ACS code to 4.7.0, why would I let the database schema > in > > >> an > > >>>>>>> older > > >>>>>>>> version? And if we want version DB schemas and application code > > >>>>>>> separately > > >>>>>>>> maintaining somehow compatibility between them, which would > bring > > a > > >>>>>> whole > > >>>>>>>> other level of complexity to the code; I think we should avoid > > >> that. > > >>>>>>>> > > >>>>>>>> The flywaydb can be easily integrated with everything we have > now; > > >> we > > >>>>>>> could > > >>>>>>>> have a maven profile for developers and integrate it in ACS > > >> bootstrap > > >>>>>>> using > > >>>>>>>> its API as a Spring bean. Therefore, we could remove the current > > >>>>>>>> “DatabaseUpgradeChecker “, “DbUpgrade” and other classes that > aim > > >> to > > >>>>> do > > >>>>>>>> that. We could even add the creation of the schema into the > first > > >>>>> time > > >>>>>> it > > >>>>>>>> boots using flywaydb and retire the “cloudstack-setup-database” > > >>>>> script, > > >>>>>>> or > > >>>>>>>> at least make it less complicated, using it just to configure > the > > >>>>>>> database > > >>>>>>>> URL and users. > > >>>>>>>> > > >>>>>>>> The point is that to use Flywaydb we would have to agree upon a > > >>>>>>> convention > > >>>>>>>> on creating routines (java and SQL) to execute upgrades. > Moreover, > > >>>>>> using > > >>>>>>> a > > >>>>>>>> tool such as Flywaydb we do not need to worry about upgrade > paths. > > >>>>> As I > > >>>>>>>> wrote in the email I used to start this thread, the upgrade has > to > > >> be > > >>>>>>>> straightforward, to go to a version we have to run all of the > > >> upgrade > > >>>>>>>> routines between the current version until the desired one. Our > > job > > >>>>> is > > >>>>>> to > > >>>>>>>> create upgrade routines that work and name them properly, the > job > > >> of > > >>>>>> the > > >>>>>>>> tool is to check the current version, the desired one, the > > upgrades > > >>>>>> that > > >>>>>>> it > > >>>>>>>> needs to run and execute everything properly. > > >>>>>>>> > > >>>>>>> > > >>>>>>> Yes, indeed. I just wanted to start the discussion if we > shouldn't > > >>>>>>> version the database differently from the code. > > >>>>>>> > > >>>>>>>> Additionally, I do not see the need to break compatibility as > > Rohit > > >>>>>>>> suggested in the FS; in my opinion, everything we have up today > > can > > >>>>> be > > >>>>>>>> migrated to the new structure I proposed. If we use a tool such > as > > >>>>>>>> Flywaydb, I even volunteered for that. The only thing we have to > > >>>>>> discuss > > >>>>>>>> and agree upon is the naming conventions for upgrades routines, > > >> where > > >>>>>> to > > >>>>>>>> put them and the configurations for flywaydb. > > >>>>>>>> > > >>>>>>>> Thanks for your contribution and time. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On Mon, Dec 28, 2015 at 2:10 PM, Rohit Yadav < > > >>>>>> rohit.ya...@shapeblue.com> > > >>>>>>>> wrote: > > >>>>>>>> > > >>>>>>>>> Hi Rafael and Wido, > > >>>>>>>>> > > >>>>>>>>> Thanks for starting a conversation in this regard, I could not > > >>>>> pursue > > >>>>>>> the > > >>>>>>>>> Chimp tool due to other $dayjob work though it’s good to see > some > > >>>>>>>>> discussion has started again. Hope we’ll solve this in 2016. > > >>>>>>>>> > > >>>>>>>>> In my opinion, we will need to first separate the database > > >>>>>>> init/migration > > >>>>>>>>> tooling away from mgmt server (right now the mgmt server does > db > > >>>>>>> migrations > > >>>>>>>>> when it starts and there is a code/db version mismatch) and > > >> secondly > > >>>>>>> make > > >>>>>>>>> sure that we’re using the same code/tool to deploy database > > (right > > >>>>>> now, > > >>>>>>>>> users use the cloudstack-setup-database python tool while > > >> developer > > >>>>>> use > > >>>>>>> the > > >>>>>>>>> maven/java DatabaseCreator activated by the -Ddeploydb flag). > > >>>>>>>>> > > >>>>>>>>> After we’ve addressed these two issues we can look into how we > > can > > >>>>>>> support > > >>>>>>>>> minor releases workflow (or decide to do something else, like > not > > >>>>>>> support > > >>>>>>>>> .Z releases like Wido mentioned), and see if we can or want to > > use > > >>>>> any > > >>>>>>>>> existing migration tool or write a wrapper tool “chimp” that > uses > > >>>>>>> existing > > >>>>>>>>> tools (some of those are mentioned in the Chimp FS like > flywaydb > > >>>>> etc). > > >>>>>>> For > > >>>>>>>>> allowing users to go back and forth from a db schema/version, > > >> we’ll > > >>>>>> also > > >>>>>>>>> need some new DB migration > > >>>>>> conventions/versioning/rules/static-checking, > > >>>>>>>>> and how developer need to write such paths (forward and > reverse) > > >>>>> etc. > > >>>>>>>>> > > >>>>>>>>> The best approach I figured at the time was to decide that > we’ll > > >> use > > >>>>>> the > > >>>>>>>>> previous db upgrade path mechanism till a certain CloudStack > > >> version > > >>>>>>> (say > > >>>>>>>>> 4.8.0) and after that we’ll use the new approach or tooling to > > >>>>>>>>> upgrade/downgrade DB schemas (thereby retiring away from the > old > > >> DB > > >>>>>>> upgrade > > >>>>>>>>> path mess). > > >>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com> Rohit Yadav > > >> Software > > >>>>>>>>> Architect , ShapeBlue d: * | s: +44 203 603 0540* > > >>>>>>>>> <%7C%20s:%20+44%20203%20603%200540> | m: *+91 8826230892* > > >>>>>>>>> <+91%208826230892> e: *rohit.ya...@shapeblue.com | t: * > > >>>>>>>>> <rohit.ya...@shapeblue.com%20%7C%20t:> | w: * > > www.shapeblue.com > > >> * > > >>>>>>>>> <http://www.shapeblue.com> a: > > >>>>>>>>> 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape Blue > Ltd > > >>>>> is a > > >>>>>>>>> company incorporated in England & Wales. ShapeBlue Services > India > > >>>>> LLP > > >>>>>>> is a > > >>>>>>>>> company incorporated in India and is operated under license > from > > >>>>> Shape > > >>>>>>> Blue > > >>>>>>>>> Ltd. Shape Blue Brasil Consultoria Ltda is a company > incorporated > > >> in > > >>>>>>> Brasil > > >>>>>>>>> and is operated under license from Shape Blue Ltd. ShapeBlue SA > > >> Pty > > >>>>>> Ltd > > >>>>>>> is > > >>>>>>>>> a company registered by The Republic of South Africa and is > > traded > > >>>>>> under > > >>>>>>>>> license from Shape Blue Ltd. ShapeBlue is a registered > trademark. > > >>>>>>>>> This email and any attachments to it may be confidential and > are > > >>>>>>> intended > > >>>>>>>>> solely for the use of the individual to whom it is addressed. > Any > > >>>>>> views > > >>>>>>> or > > >>>>>>>>> opinions expressed are solely those of the author and do not > > >>>>>> necessarily > > >>>>>>>>> represent those of Shape Blue Ltd or related companies. If you > > are > > >>>>> not > > >>>>>>> the > > >>>>>>>>> intended recipient of this email, you must neither take any > > action > > >>>>>> based > > >>>>>>>>> upon its contents, nor copy or show it to anyone. Please > contact > > >> the > > >>>>>>> sender > > >>>>>>>>> if you believe you have received this email in error. > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> On 28-Dec-2015, at 9:10 PM, Wido den Hollander <w...@widodh.nl > > > > >>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> On 28-12-15 16:21, Rafael Weingärtner wrote: > > >>>>>>>>>>> Thanks for your contribution Wido, > > >>>>>>>>>>> I have not seen Rohit’s email; I will take a look at it. > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> Ok, he has a FS here: > > >>>>>>>>>> > > >>>>>>> > > >>>>> > > >> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+Chimp > > >>>>>>>>>> > > >>>>>>>>>>> About database schema changes happening only in X.Y, I also > > >> agree > > >>>>>> with > > >>>>>>>>> you > > >>>>>>>>>>> (that is a convention we all could agree on, and such as > > conding > > >>>>> and > > >>>>>>>>>>> release procedures we could have a wiki page for that). > > >> However, I > > >>>>>>>>> think we > > >>>>>>>>>>> still might have scripts in versions X.Y.Z to add data to a > > >> table > > >>>>>> such > > >>>>>>>>> as > > >>>>>>>>>>> “guest_os_hypervisor”. > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> Yes, that is true. A bugfix could be a addition into the > > >> database, > > >>>>>> but > > >>>>>>>>>> we have to prevent it as much as possible. > > >>>>>>>>>> > > >>>>>>>>>>> The point to manage such scripts is that, if we are in > version > > >>>>> such > > >>>>>> as > > >>>>>>>>>>> 4.7.0 and a new script emerges in version 4.5.3, we would > have > > >> to > > >>>>>>>>> decide to > > >>>>>>>>>>> run or not to run it. I would rather not run them, since if > > they > > >>>>> add > > >>>>>>>>>>> something to the code base; those changes should also be > > applied > > >>>>>> into > > >>>>>>>>>>> master and as a consequence it will be available in a future > > >>>>> update. > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> I understand, but this is where our release cycle becomes the > > >>>>>> problem. > > >>>>>>>>>> It is because we release a X.Y.Z release we run into these > kind > > >> of > > >>>>>>>>> problems. > > >>>>>>>>>> > > >>>>>>>>>> If we as a project simple do not release the .Z releases we > > would > > >>>>> be > > >>>>>>>>>> fine as well ;) > > >>>>>>>>>> > > >>>>>>>>>> You can try to complicate things with technical things, or if > we > > >>>>>>> release > > >>>>>>>>>> every two / three weeks we don't run into these kind of > > >> situations > > >>>>> :) > > >>>>>>>>>> > > >>>>>>>>>> We might even cut the database version loose from the code > > >> version. > > >>>>>>>>>> > > >>>>>>>>>> Database version is simple 100, 101, 102, 103, 104, 105. And a > > >> code > > >>>>>>>>>> version requires a certain version of the database. > > >>>>>>>>>> > > >>>>>>>>>> Wido > > >>>>>>>>>> > > >>>>>>>>>>> On Mon, Dec 28, 2015 at 12:50 PM, Wido den Hollander < > > >>>>>> w...@widodh.nl> > > >>>>>>>>> wrote: > > >>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> On 28-12-15 14:16, Rafael Weingärtner wrote: > > >>>>>>>>>>>>> Hi all devs, > > >>>>>>>>>>>>> First of all, sorry the long text, but I hope we can start > a > > >>>>>>>>> discussion > > >>>>>>>>>>>>> here and improve that part of ACS. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> A while ago I have faced the code that Apache CloudStack > > (ACS) > > >>>>>> uses > > >>>>>>> to > > >>>>>>>>>>>>> upgrade from a version to newer one and that did not seem > to > > >> be > > >>>>> a > > >>>>>>> good > > >>>>>>>>>>>> way > > >>>>>>>>>>>>> to execute our upgrades. Therefore, I decided to use some > > time > > >>>>> to > > >>>>>>>>> search > > >>>>>>>>>>>>> for alternatives. > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> I think we all saw that happen once or more :) > > >>>>>>>>>>>> > > >>>>>>>>>>>>> I have read some material about versioning of scripts used > to > > >>>>>>> upgrade > > >>>>>>>>> a > > >>>>>>>>>>>>> database (DB) of a system and went through some frameworks > > >> that > > >>>>>>> could > > >>>>>>>>>>>> help > > >>>>>>>>>>>>> us. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> In the literature of software engineering, it is firmly > > stated > > >>>>>> that > > >>>>>>> we > > >>>>>>>>>>>> have > > >>>>>>>>>>>>> to version DB scripts as we do with the source code of the > > >>>>>>>>> application, > > >>>>>>>>>>>>> using the baseline approach. Gladly, we were not that bad > at > > >>>>> this > > >>>>>>>>> point, > > >>>>>>>>>>>> we > > >>>>>>>>>>>>> already versioned our routines for DB upgrade (.sql and > > >> .java). > > >>>>>>>>>>>> Therefore, > > >>>>>>>>>>>>> it seemed that we just did not have used a practical > approach > > >> to > > >>>>>>> help > > >>>>>>>>> us > > >>>>>>>>>>>>> during DB upgrades. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> From my readings and looking at the ACS source code I > raised > > >> the > > >>>>>>>>>>>> following > > >>>>>>>>>>>>> requirement: > > >>>>>>>>>>>>> • We should be able to write more than one routine to > upgrade > > >>>>> to a > > >>>>>>>>>>>>> version; those routines can be written in Java and SQL. We > > >> might > > >>>>>>> have > > >>>>>>>>>>>> more > > >>>>>>>>>>>>> than a routine to be executed for each version and we > should > > >> be > > >>>>>> able > > >>>>>>>>> to > > >>>>>>>>>>>>> define an order of execution. Additionally, to go to an > upper > > >>>>>>>>> version, we > > >>>>>>>>>>>>> have to run all of the routines from smaller versions > first, > > >>>>> until > > >>>>>>> we > > >>>>>>>>>>>>> achieve the desired version. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> We could also add another requirement that is the downgrade > > >>>>> from a > > >>>>>>>>>>>> version, > > >>>>>>>>>>>>> which we currently do not support. With that comes my first > > >>>>>> question > > >>>>>>>>> for > > >>>>>>>>>>>>> discussion: > > >>>>>>>>>>>>> • Do we want/need a method to downgrade from a version to a > > >>>>>> previous > > >>>>>>>>>>>> one? > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> I personally do not care. Usually people should create a > > backup > > >>>>>> PRIOR > > >>>>>>>>> to > > >>>>>>>>>>>> a upgrade. If that fails they can restore the backup. > > >>>>>>>>>>>> > > >>>>>>>>>>>>> I found an explanation for not supporting downgrades, and I > > >>>>> liked > > >>>>>>> it: > > >>>>>>>>>>>>> http://flywaydb.org/documentation/faq.html#downgrade > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> So, what I devised for us: > > >>>>>>>>>>>>> First the bureaucracy part - our migrations occur basically > > in > > >>>>>> three > > >>>>>>>>> (3) > > >>>>>>>>>>>>> steps, first we have a "prepare script", then a cleanup > > script > > >>>>> and > > >>>>>>>>>>>> finally > > >>>>>>>>>>>>> the migration per se that is written in Java, at least, > that > > >> is > > >>>>>> what > > >>>>>>>>> we > > >>>>>>>>>>>> can > > >>>>>>>>>>>>> expect when reading the interface > > >>>>>> “com.cloud.upgrade.dao.DbUpgrade”. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Additionally, our scripts have the following naming > > >> convention: > > >>>>>>>>>>>>> schema-<currentVersion>to<desiredVersion>, which in IMHO > may > > >>>>> cause > > >>>>>>>>> some > > >>>>>>>>>>>>> confusion because at first sight we may think that from the > > >> same > > >>>>>>>>> version > > >>>>>>>>>>>> we > > >>>>>>>>>>>>> could have different paths to an upper version, which in > > >>>>> practice > > >>>>>> is > > >>>>>>>>> not > > >>>>>>>>>>>>> happening. Instead of a <currentVersion>to<version> we > could > > >>>>>> simply > > >>>>>>>>> use > > >>>>>>>>>>>>> V_<numberOfVersion>_<sequencial>.<fileExtension>, giving > > that, > > >>>>> we > > >>>>>>>>> have to > > >>>>>>>>>>>>> execute all of the V_<version> scripts that are smaller > than > > >> the > > >>>>>>>>> version > > >>>>>>>>>>>> we > > >>>>>>>>>>>>> want to upgrade. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> To clarify what I am saying, I will use an example. Let’s > say > > >> we > > >>>>>>> have > > >>>>>>>>>>>> just > > >>>>>>>>>>>>> installed ACS and ran the cloudstack-setup-database. That > > >>>>> command > > >>>>>>> will > > >>>>>>>>>>>>> create a database schema in version 4.0.0. To upgrade that > > >>>>> schema > > >>>>>> to > > >>>>>>>>>>>>> version 4.3.0 (it is just an example, it could be any other > > >>>>>>> version), > > >>>>>>>>> ACS > > >>>>>>>>>>>>> will use the following mapping: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> _upgradeMap.put("4.0.0", new DbUpgrade[] {new > > Upgrade40to41(), > > >>>>> new > > >>>>>>>>>>>>> Upgrade410to420(), new Upgrade420to421(), new > > >> Upgrade421to430()) > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> After loading the mapping, ACS will execute the scripts > > >> defined > > >>>>> in > > >>>>>>>>> each > > >>>>>>>>>>>> one > > >>>>>>>>>>>>> of the Upgrade path classes and the migration code per se. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Now, let’s say we change the “.sql” scripts name to the > > >> pattern > > >>>>> I > > >>>>>>>>>>>>> mentioned, we would have the following scripts; those are > the > > >>>>>>> scripts > > >>>>>>>>>>>> found > > >>>>>>>>>>>>> that aim to upgrade to versions between the interval 4.0.0 > – > > >>>>> 4.3.0 > > >>>>>>>>>>>>> (considering 4.3.0, since that is the goal version): > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> - schema-40to410, can be named to: V_410_A.sql > > >>>>>>>>>>>>> - schema-40to410-cleanup, can be named to: V_410_B.sql > > >>>>>>>>>>>>> - schema-410to420, can be named to: V_420_A.sql > > >>>>>>>>>>>>> - schema-410to420-cleanup , can be named to: V_420_b.sql > > >>>>>>>>>>>>> - schema-420to421, can be named to: V_421_A.sql > > >>>>>>>>>>>>> - schema-421to430, can be named to: V_430_A.sql > > >>>>>>>>>>>>> - schema-421to430-cleanup, can be named to: V_430_B.sql > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Additionally, all of the java code would have to follow the > > >> same > > >>>>>>>>>>>>> convention. For instance, we have > > >>>>>>>>> “com.cloud.upgrade.dao.Upgrade40to41”, > > >>>>>>>>>>>>> which has some java code to migrate from 4.0.0 to 4.1.0. > The > > >>>>> idea > > >>>>>> is > > >>>>>>>>> to > > >>>>>>>>>>>>> extract that migration code to a Java class named: > > >> V_410_C.java, > > >>>>>>>>> giving > > >>>>>>>>>>>>> that it has to execute the SQL scripts before the java > code. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> In order to go from a smaller version (4.0.0) to an upper > one > > >>>>>>>>> (4.3.0), we > > >>>>>>>>>>>>> have to run all of the migration routines from intermediate > > >>>>>>> versions. > > >>>>>>>>>>>> That > > >>>>>>>>>>>>> is what we are already doing, but we do all of that > manually. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Bottom line, I think we could simple use the convention > > >>>>>>>>>>>>> V_<numberOfVersion>_<sequencial>.<fileExtension> to name > > >> upgrade > > >>>>>>>>>>>> routines. > > >>>>>>>>>>>>> That would facilitate us to use a framework to help us with > > >> that > > >>>>>>>>> process. > > >>>>>>>>>>>>> Additionally, I believe that we should always assume that > to > > >> go > > >>>>>>> from a > > >>>>>>>>>>>>> smaller version to a higher one, we should run all of the > > >>>>> scripts > > >>>>>>> that > > >>>>>>>>>>>>> exist between them. What do you guys think of that? > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> That seems good to me. But we still have to prevent that we > > >>>>> perform > > >>>>>>>>>>>> database changes in a X.Y.Z release since that is branched > off > > >>>>> to a > > >>>>>>>>>>>> different branch. > > >>>>>>>>>>>> > > >>>>>>>>>>>> Imho database changes should only happen in X.Y releases. > > >>>>>>>>>>>> > > >>>>>>>>>>>>> After the bureaucracy, we can discuss tools. If we use that > > >>>>>>>>> convention to > > >>>>>>>>>>>>> name migration (upgrade) routines, we can start thinking on > > >>>>> tools > > >>>>>> to > > >>>>>>>>>>>>> support our migration process. I found two (2) promising > > ones: > > >>>>>>>>> Liquibase > > >>>>>>>>>>>>> and Flywaydb (both seem to be under Apache license, but the > > >>>>> first > > >>>>>>> one > > >>>>>>>>> has > > >>>>>>>>>>>>> an enterprise version?!). After reading the documentation > and > > >>>>> some > > >>>>>>>>> usage > > >>>>>>>>>>>>> examples I found the flywaydb easier and simpler to use. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> What are the options of tools that we can use to help us > > >> manage > > >>>>>> the > > >>>>>>>>>>>>> database upgrade, without needing to code the upgrade path > > >> that > > >>>>>> you > > >>>>>>>>> know? > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> After that, I think we should decide if we should create > > >> another > > >>>>>>>>>>>>> project/component to take care of migrations, or we can > just > > >> add > > >>>>>> the > > >>>>>>>>>>>>> dependency of the tool to a project such as > > >> “cloud-framework-db” > > >>>>>> and > > >>>>>>>>>>>> start > > >>>>>>>>>>>>> using it. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> The “cloud-framework-db” project seems to have a focus on > > >> other > > >>>>>>> things > > >>>>>>>>>>>> such > > >>>>>>>>>>>>> as managing transactions and generating SQLs from > annotations > > >>>>> (?!? > > >>>>>>>>> That > > >>>>>>>>>>>>> should be a topic for another discussion). Therefore, I > would > > >>>>>> rather > > >>>>>>>>>>>> create > > >>>>>>>>>>>>> a new project that has the specific goal of managing ACS DB > > >>>>>>> upgrades. > > >>>>>>>>> I > > >>>>>>>>>>>>> would also move all of the routines (SQL and Java) to this > > new > > >>>>>>>>> project. > > >>>>>>>>>>>>> This project would be a module of the CloudStack project > and > > >> it > > >>>>>>> would > > >>>>>>>>>>>>> execute the upgrade routines at the startup of ACS. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> I believe that going from a homemade solution to one that > is > > >>>>> more > > >>>>>>>>>>>>> consolidated and used by other communities would be the way > > to > > >>>>> go. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> I can volunteer myself to create a PR with the > aforementioned > > >>>>>>> changes > > >>>>>>>>> and > > >>>>>>>>>>>>> using flywaydb to manage our upgrades. However, I prefer to > > >>>>> have a > > >>>>>>>>> good > > >>>>>>>>>>>>> discussion with other devs first, before starting coding. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Do you have suggestions or points that should be raised > > before > > >>>>> we > > >>>>>>>>> start > > >>>>>>>>>>>>> working on that? > > >>>>>>>>>>>> > > >>>>>>>>>>>> Rohit suggested Chimp earlier this year: > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>> > > >> > > > http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201508.mbox/%3c677bd09f-fc75-4888-8dc8-2b7af7439...@shapeblue.com%3E > > >>>>>>>>>>>> > > >>>>>>>>>>>> The thread is called: "[DISCUSS] Let's fix CloudStack > Upgrades > > >>>>> and > > >>>>>> DB > > >>>>>>>>>>>> migrations with CloudStack Chimp" > > >>>>>>>>>>>> > > >>>>>>>>>>>> Maybe there is something good in there. > > >>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> -- > > >>>>>>>>>>>>> Rafael Weingärtner > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> Regards. > > >>>>>>>>> > > >>>>>>>>> Find out more about ShapeBlue and our range of CloudStack > related > > >>>>>>> services: > > >>>>>>>>> IaaS Cloud Design & Build > > >>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> | > CSForge – > > >>>>>> rapid > > >>>>>>>>> IaaS deployment framework <http://shapeblue.com/csforge/> > > >>>>>>>>> CloudStack Consulting < > > >> http://shapeblue.com/cloudstack-consultancy/ > > >>>>>> > > >>>>>> | > > >>>>>>> CloudStack > > >>>>>>>>> Software Engineering > > >>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/> > > >>>>>>>>> CloudStack Infrastructure Support > > >>>>>>>>> <http://shapeblue.com/cloudstack-infrastructure-support/> | > > >>>>>> CloudStack > > >>>>>>>>> Bootcamp Training Courses < > > >>>>> http://shapeblue.com/cloudstack-training/> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> -- > > >>>>>> Rafael Weingärtner > > >>>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> -- > > >>>>> Daan > > >>>>> > > >>>> > > >>>> > > >>>> > > >>> > > >> > > >> > > >> > > >> -- > > >> Rafael Weingärtner > > >> > > > > > > > > > > > > > > > -- > Rafael Weingärtner > -- Daan
