> You still can do it with file per release. >>> Regardless of the one per release argument vs many files. We still haven't >>> created fuel_4.1.py which if we are doing one per release, is very >>> necessary. There is no point of managing db migrations if we don't create >>> the files per release.
> While we don't have upgrades we need to have a single migrations file i.e. > 5.0. > When we start develop 5.1 release we will create 5.1 migration file. Again, we still do not have a 4.1 migration file which should have been created already. So we aren't even creating one file per release as you indicated we are. On Tue, Apr 15, 2014 at 10:10 AM, Evgeniy L <[email protected]> wrote: > Hi guys, sorry, but I was really busy and didn't have time to respond you > >>> I have also never seen database migrations handled in any other way than >>> with multiple files. > > We will have multiple files, but it will be single file per release. > >>> I agree with Nikolay and Ryan, Multiple files makes more sense. One >>> Alembic tracks the dependencies between them and applies them in order. Two >>> it allow us to revert changes that include db changes safely. Three it >>> allows people with working db's to migrate between versions safely between >>> revisions. > > You still can do it with file per release. > >>> Regardless of the one per release argument vs many files. We still >>> haven't created fuel_4.1.py which if we are doing one per release, is very >>> necessary. There is no point of managing db migrations if we don't create >>> the files per release. > > While we don't have upgrades we need to have a single migrations file i.e. > 5.0. > When we start develop 5.1 release we will create 5.1 migration file. > > I'll try to to describe problems which we will have in case of several files > per release. > > 1. we have to create some kind of naming convention for this files. And > there will be a lot -1 for this. > E.g. > 330ec2ab2bbf_add_nodegroup.py - where nodegroup was added? > 596b7e3f2b11_upgrade.py - or this file's name tells almost nothing > > In case of file per release it much simpler > > fuel_5_0.py > fuel_5_1.py > > 2. from this file names it's not obvious what order will be used to apply > this migrations files, you need to run some script to find the order. > > In case of file per release it is obvious what order will be used > > fuel_5_0.py > fuel_5_1.py > > 3. and argument from my previous email > > Developer A added field "a". > Developer B during development found that this field and decided to delete > it or to rename it. > > 4_0_fix_project_user_quotas_resource_length.py > 4_0_add_a_in_compute_nodes.py - Developer A added this migration file > 4_0_add_extra_resources_in_compute_nodes.py > 4_0_add_details_column_to_instance_actions_events.py > 4_0_add_ephemeral_key_uuid.py - Last migration > > What developer B should to do? Should he create new > migration file or should he change/remove previous files? > It's very easy to miss the file '4_0_add_a_in_compute_nodes.py' > in the list, in this case developer will create new extra migration > file to remove or to rename field "a". > > In case of single migration file per release developer will be able > to see, that this field was added in the current release, and > he will be able to remove/rename it. > > [0] > https://github.com/stackforge/fuel-web/tree/master/nailgun/nailgun/db/migration/alembic_migrations/versions > > Thanks, > > > On Sat, Apr 12, 2014 at 1:25 AM, Andrew Woodward <[email protected]> wrote: >> >> Ryan helped to find that the changes I found from [1] are in fact due to >> buggy migrations from Alembic 0.6.2, moving to 0.6.4 resolves this issue. So >> that was a false alarm. I am intrigued as to why no one has raised this. >> >> [1] https://gist.github.com/xarses/10498338 >> >> >> On Fri, Apr 11, 2014 at 1:21 PM, Andrew Woodward <[email protected]> wrote: >>> >>> I agree with Nikolay and Ryan, Multiple files makes more sense. One >>> Alembic tracks the dependencies between them and applies them in order. Two >>> it allow us to revert changes that include db changes safely. Three it >>> allows people with working db's to migrate between versions safely between >>> revisions. >>> >>> Regardless of the one per release argument vs many files. We still >>> haven't created fuel_4.1.py which if we are doing one per release, is very >>> necessary. There is no point of managing db migrations if we don't create >>> the files per release. >>> >>> Also, I have found that there are changes currently in master that are >>> not covered by a migration [1]. This shows that either changes aren't being >>> tracked propery in with current.py or people don't what or how to update >>> this. If we are going to keep the one-per-release approach, it would be >>> better to just not manage the migration files until we are ready to generate >>> the release and create it once. >>> >>> [1] https://gist.github.com/xarses/10498338 >>> >>> >>> On Fri, Apr 11, 2014 at 12:28 PM, Ryan Moe <[email protected]> wrote: >>>> >>>> Have we reached a consensus on how we're handling migrations? I see some >>>> reviews modifying current.py and some adding new migration files. FWIW I >>>> agree with everything Nikolay said. I have also never seen database >>>> migrations handled in any other way than with multiple files. >>>> >>>> Thanks, >>>> Ryan >>>> >>>> On Mon, Mar 31, 2014 at 7:12 AM, Nikolay Markov <[email protected]> >>>> wrote: >>>>>> >>>>>> I think it will be easier to add changes in a single >>>>>> schema instead of merging before release because >>>>>> in case of merging we have additional manual >>>>>> labour, we need to remember that we need to do it >>>>>> before release and we need to merge the migration >>>>>> files manually. >>>>> >>>>> >>>>> All we need to do in this case is simple copy-paste, it can even be >>>>> automated if we are not happy about doing it by hands. All code in >>>>> upgrade() >>>>> and downgrade() methods executes one migration by one, it doesn't matter >>>>> if >>>>> it's located in one file or multiple. >>>>> >>>>>> Common practice is to keep in a single migration >>>>>> file all changes which were made during development >>>>>> cycle. >>>>> >>>>> >>>>> As long-time web developer in the past - never saw this practice. It >>>>> was always multiple files. >>>>> >>>>> I would say you're thinking too much about developers looking through >>>>> migrations. I can say you almost never need to look at previous >>>>> migrations, >>>>> you just need to create yours from previous state (no matter what it is) >>>>> to >>>>> yours. >>>>> >>>>> Also, it actually doesn't matter how long does it take to apply DB >>>>> migration. In the scope of upgrading process as a whole it will be a tiny >>>>> thing and even if we add field and then delete it - it doesn't make any >>>>> notable difference for users, but it's easier for developers to not look >>>>> back. >>>>> >>>>>> If release == new database, we will have performance degradation in N >>>>>> times (where N equal to amount of releases). >>>>> >>>>> >>>>> Why? We can do requests in parallel. And what are possible problems >>>>> with transactions? We still keep all the objects with v1 in DBv1 and >>>>> objects >>>>> v2 in DBv2. They will never intersect, in transactions as well. >>>>> >>>>> On Mon, Mar 31, 2014 at 3:28 PM, Evgeniy L <[email protected]> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> >> The question is, do we need to keep it single during development >>>>>> >> process or we should just merge all the files into one migration just >>>>>> >> before >>>>>> >> release? >>>>>> >>>>>> I think it will be easier to add changes in a single >>>>>> schema instead of merging before release because >>>>>> in case of merging we have additional manual >>>>>> labour, we need to remember that we need to do it >>>>>> before release and we need to merge the migration >>>>>> files manually. >>>>>> >>>>>> >> As for me, I don't see any issues with keeping multiple migrations >>>>>> >> in code repo (that's the common practice of majority of projects). >>>>>> >> Please >>>>>> >> write your objections. >>>>>> >>>>>> Common practice is to keep in a single migration >>>>>> file all changes which were made during development >>>>>> cycle. Our development cycles are much longer >>>>>> than development cycles of regular web services >>>>>> (it's a specific of our product) as result our migration >>>>>> files bigger. >>>>>> >>>>>> I can provide several examples why 1 migration file >>>>>> per release is better than hundreds of small migration files. >>>>>> >>>>>> 1. it looks better to have a single file per release >>>>>> >>>>>> current.py # I think we need to rename it to 5.0 >>>>>> fuel_4_0.py >>>>>> >>>>>> If you want to see what was changed between two >>>>>> versions you can just open a single file. >>>>>> >>>>>> .... here a lot of files >>>>>> 4_0_fix_project_user_quotas_resource_length.py >>>>>> 4_0_add_metrics_in_compute_nodes.py >>>>>> 4_0_add_extra_resources_in_compute_nodes.py >>>>>> 4_0_add_details_column_to_instance_actions_events.py >>>>>> 4_0_add_ephemeral_key_uuid.py >>>>>> 4_0_drop_dump_tables.py >>>>>> 4_0_add_stats_in_compute_nodes.py >>>>>> >>>>>> Here you have to follow some additional file naming >>>>>> convention. >>>>>> And not all of this names are obvious, as result you >>>>>> have to look inside of this files anyway. >>>>>> >>>>>> 2. development >>>>>> >>>>>> Developer A added field "a". >>>>>> Developer B during development found that this field and decided to >>>>>> delete it or to rename it. >>>>>> >>>>>> 4_0_fix_project_user_quotas_resource_length.py >>>>>> 4_0_add_a_in_compute_nodes.py - Developer A added this migration file >>>>>> 4_0_add_extra_resources_in_compute_nodes.py >>>>>> 4_0_add_details_column_to_instance_actions_events.py >>>>>> 4_0_add_ephemeral_key_uuid.py - Last migration >>>>>> >>>>>> What developer B should to do? Should he create new >>>>>> migration file or should he change/remove previous files? >>>>>> It's very easy to miss the file '4_0_add_a_in_compute_nodes.py' >>>>>> in the list, in this case developer will create new extra migration >>>>>> file to remove or to rename field "a". >>>>>> >>>>>> In case of single migration file per release developer will be able >>>>>> to see, that this field was added in the current release, and >>>>>> he will be able to remove/rename it. >>>>>> >>>>>> >> I proposed to use separate DB for each major API version (which may >>>>>> >> have completely independent schemas) and just write data migration >>>>>> >> scripts >>>>>> >> (v1->v2 and v2->v1), for example, to allow adding nodes to v1 cluster. >>>>>> >>>>>> If release == new database, we will have performance degradation in N >>>>>> times (where N equal to amount of releases). >>>>>> How are you going to use transactions when you have several databases? >>>>>> It adds complexity. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Mar 28, 2014 at 7:12 PM, Nikolay Markov <[email protected]> >>>>>> wrote: >>>>>>> >>>>>>> Hello colleagues, >>>>>>> >>>>>>> Right now we already have working DB migration mechanism presented by >>>>>>> Alembic, but it becomes more and more complex as we move towards >>>>>>> upgrades. >>>>>>> >>>>>>> First, as we agreed, migration from previous version of Fuel DB to >>>>>>> the >>>>>>> next one will be presented by a single file. The question is, do we >>>>>>> need to keep it single during development process or we shouls just >>>>>>> merge all the files into one migration just before release? >>>>>>> >>>>>>> To clarify things, it's not really possible to generate completely >>>>>>> working migration from the scratch taking the diff between two >>>>>>> releases, because there are some issues in auto-generated scripts >>>>>>> which may be fixed by hands only during development. And our single >>>>>>> migration script (current.py) is becoming more and more huge as we >>>>>>> don't keep small updates in a separate files. >>>>>>> >>>>>>> As for me, I don't see any issues with keeping multiple migrations in >>>>>>> code repo (that's the common practice of majority of projects). >>>>>>> Please >>>>>>> write your objections. >>>>>>> >>>>>>> Second, it's not clear right now how we're going to achieve backward >>>>>>> compatibility. We will have separate versions of almost all objects >>>>>>> in >>>>>>> code and will select corresponding ones by Environment versions. The >>>>>>> thing is, it will be very hard for us to write working migrations in >>>>>>> both directions without serious data loss, especially if we'll have >>>>>>> lots of changes in DB schema. >>>>>>> >>>>>>> I proposed to use separate DB for each major API version (which may >>>>>>> have completely independent schemas) and just write data migration >>>>>>> scripts (v1->v2 and v2->v1), for example, to allow adding nodes to v1 >>>>>>> cluster. This seems as a huge overhead, but actually helps to get >>>>>>> away >>>>>>> of bad headache writing DB migrations. >>>>>>> >>>>>>> Please let's discuss all these things it this thread. >>>>>>> >>>>>>> -- >>>>>>> Best regards, >>>>>>> Nick Markov >>>>>>> >>>>>>> -- >>>>>>> Mailing list: https://launchpad.net/~fuel-dev >>>>>>> Post to : [email protected] >>>>>>> Unsubscribe : https://launchpad.net/~fuel-dev >>>>>>> More help : https://help.launchpad.net/ListHelp >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Best regards, >>>>> Nick Markov >>>>> >>>>> -- >>>>> Mailing list: https://launchpad.net/~fuel-dev >>>>> Post to : [email protected] >>>>> Unsubscribe : https://launchpad.net/~fuel-dev >>>>> More help : https://help.launchpad.net/ListHelp >>>>> >>>> >>>> >>>> -- >>>> Mailing list: https://launchpad.net/~fuel-dev >>>> Post to : [email protected] >>>> Unsubscribe : https://launchpad.net/~fuel-dev >>>> More help : https://help.launchpad.net/ListHelp >>>> >>> >>> >>> >>> -- >>> Andrew >>> Mirantis >>> Ceph community >> >> >> >> >> -- >> Andrew >> Mirantis >> Ceph community > > -- Andrew Mirantis Ceph community -- Mailing list: https://launchpad.net/~fuel-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~fuel-dev More help : https://help.launchpad.net/ListHelp
