> -----Original Message----- > From: rohityada...@gmail.com [mailto:rohityada...@gmail.com] On Behalf > Of Rohit Yadav > Sent: Tuesday, February 19, 2013 4:16 PM > To: cloudstack-dev@incubator.apache.org > Cc: Alex Huang > Subject: Re: Merge 4.1-new-db-schema.sql to db/db/schema-40to410.sql > > On Mon, Feb 18, 2013 at 2:30 PM, Koushik Das <koushik....@citrix.com> > wrote: > > Rohit, > > > > Currently I see that there are 2 sql files related to upgrade from version A > to B, for e.g. schema-40to410.sql and schema-40to410-cleanup.sql. > > 1. What all should go in the normal one and what all in the -cleanup one? > For e.g. should deleting table metadata go in cleanup script, should deleting > table data go in cleanup script? > > In the schema one you should keep rules like add columns, add tables, > copying data (like copying to uuid column from id) etc. In the cleanup script > you should do things like removing columns or data. Think in this way, when > multiple mgmt servers are running and not all of them are on same version, > we want the first step of database upgrade to be backward compatible to > the old one which means you're not removing any column, after you've > upgraded all mgmt server you are free to cleanup and remove any column, > copy any data from those columns to maybe new ones. >
[Koushik] So just to summarize the upgrade process from what you have described. 1. The db is first upgraded with new additions (adding a new metadata, adding new data). 2. Individual MSs are upgraded. 3. All the db cleanup scripts are executed where delete/edits are done. Now some issues I see with the above: Running step 1 above may result in problems as the MSs are still running and probably doing some changes in the db. What happens if the step 1 fails midway - will partial changes be persisted or they will be reverted. In former case if the upgrade process is repeated the scripts needs to be able to handle the case where something already exists. > > 2. What is the order in which the scripts will be executed? Upgrade > followed by corresponding cleanup for each version? OR is it like call all > upgrades for all versions followed by all cleanup scripts? > > At present, all create schemas are called first and cleanup is done only at > the > end. We'll have the same way for 4.1 but starting 4.2, once we will have the > DatabaseCreator tool, it would much easier and sysadmins could decide how > they want to upgrade. Alex can comment on it. > > > 3. If the ordering is based on former case (in 2 above) then it is better to > combine them into one single sql script. If it is the latter then I would > like to > understand the reason behind that? > > Read explanation in 1. In case of clustered mgmt servers, if an old mgmt > server is running they will die if you remove any column of some table it can > access. While one may argue why not bring down the whole cloud and then > do upgrades, which is fine (and will be possible with > dbcreator/4.2) but not possible as only mgmt server upgrades the db now. > > Regards. > > > > > Thanks, > > Koushik > > > >> -----Original Message----- > >> From: rohityada...@gmail.com [mailto:rohityada...@gmail.com] On > >> Behalf Of Rohit Yadav > >> Sent: Saturday, February 16, 2013 5:51 PM > >> To: cloudstack-dev@incubator.apache.org; Edison Su > >> Cc: Sheng Yang; Alex Huang; Kishan Kavala; Min Chen > >> Subject: Re: Merge 4.1-new-db-schema.sql to db/db/schema-40to410.sql > >> > >> I've applied my changes on master, to gain visibility of those > >> changes and so community can grok I started a new thread of db > >> changes for 4.1 to support rolling db upgrades. > >> > >> Regards. > >> > >> On Sat, Feb 16, 2013 at 1:58 PM, Rohit Yadav <bhais...@apache.org> > wrote: > >> > On Fri, Feb 15, 2013 at 11:26 PM, Edison Su <edison...@citrix.com> > wrote: > >> >> > >> >> > >> >>> -----Original Message----- > >> >>> From: rohityada...@gmail.com [mailto:rohityada...@gmail.com] On > >> >>> Behalf Of Rohit Yadav > >> >>> Sent: Friday, February 15, 2013 2:58 AM > >> >>> To: Edison Su; cloudstack-dev@incubator.apache.org > >> >>> Cc: Sheng Yang; Alex Huang; Kishan Kavala; Min Chen > >> >>> Subject: Merge 4.1-new-db-schema.sql to db/db/schema- > 40to410.sql > >> >>> > >> >>> Edison, you'd created a 4.1-new-db-schema sql, may I merge > >> >>> 4.1-new-db- schema.sql to db/db/schema-40to410.sql? We'll also > >> >>> need a db/db/schema- 40to410-cleanup.sql and upgrading class. > >> >> > >> >> Yes, it should be merged into schema-40to410.sql. > >> > > >> > Hi Edison, there was one merge issue while moving the extra create > >> > rules from create-schema to schema-40to410.sql. In table > >> > cloud.template_s3_ref; > >> > > >> > diff --git a/setup/db/db/schema-40to410.sql > >> > b/setup/db/db/schema-40to410.sql index 9a9441d..3fb1e32 100644 > >> > --- a/setup/db/db/schema-40to410.sql > >> > +++ b/setup/db/db/schema-40to410.sql > >> > @@ -171,7 +171,7 @@ CREATE TABLE `cloud`.`template_s3_ref` ( > >> > CONSTRAINT `fk_template_s3_ref__template_id` FOREIGN KEY > >> > `fk_template_s3_ref__template_id` (`template_id`) REFERENCES > >> > `vm_template` (`id`), > >> > - INDEX `i_template_s3_ref__swift_id`(`s3_id`), > >> > + INDEX `i_template_s3_ref__s3_id`(`s3_id`), > >> > INDEX `i_template_s3_ref__template_id`(`template_id`) > >> > ) ENGINE=InnoDB DEFAULT CHARSET=utf8; > >> > > >> > I chose the change in create-schema, not not in schema-40to410.sql. > >> > Please check if it's okay, or fix as needed. Thanks. > >> > > >> > > >> > > >> >> > >> >>> > >> >>> There is some difference between the 4.0 create-schema.sql and > >> >>> that in > >> >>> 4.1: > >> >>> http://people.apache.org/~bhaisaab/diff-create-schema-40-41.sql > >> >>> > >> >>> Alex suggested in last discussion that we should not change the > >> >>> schema file and have upgrade paths and cleanup paths. This would > >> >>> help us do rolling upgrades. But Alex, there are a lot of > >> >>> difference between the > >> >>> 4.0 schema and 4.1 schema sqls (about 4k lines of diff): > >> >>> http://people.apache.org/~bhaisaab/diff4041sqls.sql If we don't > >> >>> want developers to change anything in the schema files, how do > >> >>> you propose we fix it now for 4.1? Move all these changes to 40to41 > sql? > >> >>> > >> >>> Sheng (ipv6), Kishan (regions), Vijay (autoscale) I see some > >> >>> create schema changes, should we move them to db/db/schema- > 40to410.sql? > >> >>> > >> >>> Min, the create-schema-view did not exist for 4.0, should we keep > >> >>> it as it as, our move it as upgrade path? > >> >>> > >> >>> For 4.1, we should do following changes so DatabaseCreator could > >> >>> be refactored and used starting 4.2/master: > >> >>> - Change version to 4.0 in create-schema > >> >>> - Make sure there is difference between 4.0 and 4.1 create-schema > >> >>> sqls > >> >>> - Move new additions to db/db/schema-40to410.sql, define a > >> >>> cleanup path/sql and upgrading class from 40-41 > >> >>> - Fix cloudstack-setup-databases, for 4.1 we won't use > >> >>> databasecreator, stick with the present script (won't risk > >> >>> regressions and whatnot) > >> >>> > >> >>> Regards.
