Hi, Paul! This messages is OK. May be you can put a change on review in WIP status, that I will be able to check what is going on? I never have such problems with migrations in neutron-vpnaas repo. May be the problem is that database is already upgraded, was database cleaned before you run neutron-db-manage upgrade head?
On Tue, Jul 7, 2015 at 11:35 PM, Paul Michali <p...@michali.net> wrote: > Well, I can run the upgrade command, but it doesn't seem to be processing > my new migration file. I have tried using upgrade with 'head', and with the > HEAD file set to the previous version and to the new version. In both > cases, I get these info messages twice: "Context impl MySWLImpl." and "Will > assume non-transactional DDL." I put a "import pdb; pdb.set_trace() in my > migration file, but it never reaches that. > > What am I possibly missing? > > Regards, > > PCM > > > On Tue, Jul 7, 2015 at 4:04 PM Paul Michali <p...@michali.net> wrote: > >> I found the issue. The upgrade is looking for config to have [database] >> section and connection definition. If I use /etc/neutron/neutron.conf, then >> the neutron-db-manage runs. >> >> >> >> On Tue, Jul 7, 2015 at 3:38 PM Paul Michali <p...@michali.net> wrote: >> >>> I have that change in the neutron repo that is being used with by this >>> neutron-vpnaas repo. >>> >>> On Tue, Jul 7, 2015 at 3:12 PM Mike Bayer <mba...@redhat.com> wrote: >>> >>>> >>>> >>>> On 7/7/15 1:28 PM, Paul Michali wrote: >>>> >>>> HEAD, head, 24f28869838b (my new file) all say the same thing. :( >>>> >>>> >>>> On Tue, Jul 7, 2015 at 12:34 PM Salvatore Orlando <sorla...@nicira.com> >>>> wrote: >>>> >>>>> possibly I was wrong in mixing up git & alembic. >>>>> It should be "upgrade head" - lowercase. >>>>> >>>>> If that doesn't work there might some other issue lurking. >>>>> >>>>> Salvatore >>>>> >>>>> On 7 July 2015 at 17:44, Paul Michali <p...@michali.net> wrote: >>>>> >>>>>> Salvatore, >>>>>> >>>>>> I changed head to the version before my new one, and then tried to >>>>>> upgrade and I see this: >>>>>> neutron-db-manage --config-file >>>>>> /opt/stack/neutron/etc/neutron.conf --service vpnaas upgrade HEAD >>>>>> Traceback (most recent call last): >>>>>> File "/usr/local/bin/neutron-db-manage", line 10, in <module> >>>>>> sys.exit(main()) >>>>>> File "/opt/stack/neutron/neutron/db/migration/cli.py", line 238, in >>>>>> main >>>>>> CONF.command.func(config, CONF.command.name) >>>>>> File "/opt/stack/neutron/neutron/db/migration/cli.py", line 105, in >>>>>> do_upgrade >>>>>> run_sanity_checks(config, revision) >>>>>> File "/opt/stack/neutron/neutron/db/migration/cli.py", line 229, in >>>>>> run_sanity_checks >>>>>> script_dir.run_env() >>>>>> File "/usr/local/lib/python2.7/dist-packages/alembic/script.py", >>>>>> line 390, in run_env >>>>>> util.load_python_file(self.dir, 'env.py') >>>>>> File "/usr/local/lib/python2.7/dist-packages/alembic/util.py", line >>>>>> 243, in load_python_file >>>>>> module = load_module_py(module_id, path) >>>>>> File "/usr/local/lib/python2.7/dist-packages/alembic/compat.py", >>>>>> line 79, in load_module_py >>>>>> mod = imp.load_source(module_id, path, fp) >>>>>> File >>>>>> "/opt/stack/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/env.py", >>>>>> line 86, in <module> >>>>>> run_migrations_online() >>>>>> File >>>>>> "/opt/stack/neutron-vpnaas/neutron_vpnaas/db/migration/alembic_migrations/env.py", >>>>>> line 67, in run_migrations_online >>>>>> engine = session.create_engine(neutron_config.database.connection) >>>>>> File >>>>>> "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", >>>>>> line 112, in create_engine >>>>>> url = sqlalchemy.engine.url.make_url(sql_connection) >>>>>> File >>>>>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py", line >>>>>> 186, in make_url >>>>>> return _parse_rfc1738_args(name_or_url) >>>>>> File >>>>>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/url.py", line >>>>>> 235, in _parse_rfc1738_args >>>>>> "Could not parse rfc1738 URL from string '%s'" % name) >>>>>> sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string >>>>>> '' >>>>>> >>>>>> Any ideas what is wrong here? >>>>>> >>>>> >>>> I'm going to guess this is the issue fixed by >>>> https://review.openstack.org/#/c/194197/ >>>> >>>> >>>> >>>> >>>> >>>> >>>>>> On Tue, Jul 7, 2015 at 10:05 AM Paul Michali <p...@michali.net> wrote: >>>>>> >>>>>>> >>>>>>> Yes, I wasn't using the --service option, so I suspect that is why >>>>>>> my down_version was wrong. In talking with Akihiro, I added a check to >>>>>>> PEP8 and made sure that it fails if head is wrong. It is: >>>>>>> <https://review.openstack.org/#/c/199082/> >>>>>>> https://review.openstack.org/#/c/199082/ (of course that failed >>>>>>> py27 - I've got to see if there was some recent breakage in vpn repo, >>>>>>> again). >>>>>>> >>>>>>> Regarding the migration, one of the new columns may be None, but >>>>>>> there must be at least one IP version entry (there is an existing test >>>>>>> in >>>>>>> VPN for using a router w/o an external IP set). Since the new code will >>>>>>> rely on these new fields, I'd like to populate them as part of the >>>>>>> migration. I think it would be more complicated to handle during >>>>>>> operation. >>>>>>> >>>>>>> Does anyone have examples of how to do queries of objects, from >>>>>>> the migration upgrade() code? >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> PCM >>>>>>> >>>>>>> On Tue, Jul 7, 2015 at 9:02 AM Akihiro Motoki <amot...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> 2015-07-07 21:39 GMT+09:00 Henry Gessau < <ges...@cisco.com> >>>>>>>> ges...@cisco.com>: >>>>>>>> >>>>>>>>> On Tue, Jul 07, 2015, Paul Michali <p...@michali.net> >>>>>>>>> <p...@michali.net> <p...@michali.net> wrote: >>>>>>>>> >>>>>>>>> Thanks Salvatore for the responses. See @PCM in-line... >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Jul 7, 2015 at 6:14 AM Salvatore Orlando < >>>>>>>>> <sorla...@nicira.com>sorla...@nicira.com> wrote: >>>>>>>>> >>>>>>>>>> Some comments inline. >>>>>>>>>> >>>>>>>>>> Salvatore >>>>>>>>>> >>>>>>>>>> On 6 July 2015 at 20:00, Paul Michali < <p...@michali.net> >>>>>>>>>> p...@michali.net> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I have some urgent requests about migration that I'm hoping to >>>>>>>>>>> get some info on. I'm working on a bug where I need to add two >>>>>>>>>>> (related) >>>>>>>>>>> fields to a table for VPNaaS. Here's the objectives related to >>>>>>>>>>> migration... >>>>>>>>>>> >>>>>>>>>>> 1) create local_v4_ip and lcoal_v6_ip fields in the vpnservice >>>>>>>>>>> table >>>>>>>>>>> 2) for each entry in the vpnservice table: >>>>>>>>>>> 2.1) Get the router.gw_port.fixed_ips list >>>>>>>>>>> 2.2) Determine the version of each fixed IP and store the >>>>>>>>>>> first of each version (if any) into the appropriate new field. >>>>>>>>>>> >>>>>>>>>>> I have created a migration file, and I changed the >>>>>>>>>>> down_revision to be the number of the revision that is the first in >>>>>>>>>>> the >>>>>>>>>>> migration chain in the VPN repo. >>>>>>>>>>> >>>>>>>>>>> Here are the many questions I have... >>>>>>>>>>> >>>>>>>>>>> When I look in the VPN repo, the HEAD file has the version >>>>>>>>>>> 'kilo', which is not the current head. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Shouldn't it the version number of the first file in the >>>>>>>>>>> migration chain? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> It should indeed. How are you generating the revision >>>>>>>>>> script? Using neutron-db-manage it should be updated automatically >>>>>>>>>> [1] >>>>>>>>>> >>>>>>>>> >>>>>>>>> @PCM I ran neutron-db-manage, when in the neutron repo, and it >>>>>>>>> assigned some version, but it was not the latest in the >>>>>>>>> neutron-vpnaas repo. >>>>>>>>> >>>>>>>>> neutron-db-manage does not handle alembic branches in separate >>>>>>>>> repos very well at all yet. I am working on updating it with >>>>>>>>> https://review.openstack.org/198524 but I have quite a lot left >>>>>>>>> to do. >>>>>>>>> >>>>>>>> >>>>>>>> Yes, at now we have implicit order of running alembic >>>>>>>> migrations. >>>>>>>> First run neutron db migration and then advanced service migrations. >>>>>>>> >>>>>>>> I do not fully understand how alembic branch mechanism works, but >>>>>>>> I think we can have a common ancestor and have multiple branches, >>>>>>>> and each branch can evolve independently. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> I checked the VPN repo and there were a chain of versions, which >>>>>>>>> I used to determine what the head should be and have set the version >>>>>>>>> accordingly. However, in the current repo, head is set to "kilo", >>>>>>>>> which >>>>>>>>> appears to be incorrect. The versions are: >>>>>>>>> >>>>>>>>> 56893333aa52 >>>>>>>>> kilo <<< HEAD >>>>>>>>> 3ea02b2a773e >>>>>>>>> start_neutron_vpnaas >>>>>>>>> None >>>>>>>>> >>>>>>>>> Ouch. That is an error, because >>>>>>>>> https://review.openstack.org/190569 should have updated HEAD but >>>>>>>>> didn't. >>>>>>>>> >>>>>>>>> The version sequence (you can see it in any devstack run) is: >>>>>>>>> >>>>>>>>> INFO [alembic.migration] Running upgrade -> >>>>>>>>> start_neutron_vpnaas, start neutron-vpnaas chain >>>>>>>>> INFO [alembic.migration] Running upgrade start_neutron_vpnaas -> >>>>>>>>> 3ea02b2a773e, add_index_tenant_id >>>>>>>>> INFO [alembic.migration] Running upgrade 3ea02b2a773e -> kilo, >>>>>>>>> kilo >>>>>>>>> INFO [alembic.migration] Running upgrade kilo -> 56893333aa52, >>>>>>>>> fix identifier map fk >>>>>>>>> >>>>>>>> >>>>>>>> It seems we don't have an appropriate check for HEAD revision >>>>>>>> in at least VPNaaS repo. >>>>>>>> Paul and I just discussed it. We need to improve the check too. >>>>>>>> >>>>>>>> >>>>>>>>> Should I do a separate commit that fixes the HEAD file, or >>>>>>>>> just fix it as part of the bug fix I'm working on. >>>>>>>>> >>>>>>>>> Yes, you should immediately submit a patch to change HEAD to >>>>>>>>> 56893333aa52. >>>>>>>>> >>>>>>>>> >>>>>>>>> BTW, at one point, after having correctly set the HEAD and >>>>>>>>> versions in my new migration file, I think I ran neutron-db-manage >>>>>>>>> check_migration, and I think it set the HEAD to my version, but it >>>>>>>>> did that >>>>>>>>> in the neutron repo, and not the VPN repo. I might have been running >>>>>>>>> from >>>>>>>>> the wrong repo? >>>>>>>>> >>>>>>>>> I working on updating the devref docs for this process. Things >>>>>>>>> have changed quite a bit with the alembic branches in separate repos. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> For my commit, I'm assuming I change the HEAD file to use my >>>>>>>>>>> migration file's version? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> You can do that manually too, yes. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I set HEAD to my migration file, and my file has a down >>>>>>>>>>> revision of the previous head's revision. If I run >>>>>>>>>>> 'neutron-db-manage >>>>>>>>>>> --config-file ../neutron/etc/neutron.conf --config-file >>>>>>>>>>> ../neutron/etc/neutron/plugins/ml2/ml2_conf.ini check_migration' >>>>>>>>>>> there is >>>>>>>>>>> no output so I guess that is OK. >>>>>>>>>>> >>>>>>>>>>> As I develop my new migration file, is there a way that I can >>>>>>>>>>> test it (running neutron-db-migration, maybe)? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> When I test migrations I usually dump the database, run the >>>>>>>>>> migration with neutron-db-manage upgrade HEAD (I think it's not >>>>>>>>>> necessary >>>>>>>>>> to specify HEAD), and restore the db from the dump if the migration >>>>>>>>>> fails. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Is there a way to run the migration file under the debugger, >>>>>>>>>>> as well (importing pdb, for example)? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The migration process is just like any python application, >>>>>>>>>> so I guess you can debug it with pdb. >>>>>>>>>> >>>>>>>>> >>>>>>>>> @PCM Ah, so use "neutron-db-manage upgrade HEAD". That was the >>>>>>>>> piece that was missing. I take it there are no specific unit tests of >>>>>>>>> the >>>>>>>>> migration files? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> In the migration, I can add the columns needed. What's the >>>>>>>>>>> best way to fill out those fields - using raw SQL queries or create >>>>>>>>>>> a >>>>>>>>>>> Session object and access the VpnService object's router object? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> If the default value for the column is not enough, and you >>>>>>>>>> need to specify a value which depends on other values in the same >>>>>>>>>> row I >>>>>>>>>> would prefer plain SQL statements, but if that become cumbersome I >>>>>>>>>> guess >>>>>>>>>> it's ok to use sqlalchemy's session. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> I see there is some op.bind() call and then engine.execute(), >>>>>>>>>>> but could use some help on the best way to extract the needed >>>>>>>>>>> queries (I >>>>>>>>>>> need to access the vpnservice's router, and then access the (Port) >>>>>>>>>>> gw_port >>>>>>>>>>> relationship, and from that access the (IPAllocation) fixed_ips >>>>>>>>>>> list). >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Perhaps you can point us to the review pages on gerrit, and >>>>>>>>>> we can provide detailed comments there. >>>>>>>>>> >>>>>>>>> >>>>>>>>> @PCM Yeah, I haven't pushed it up yet. I have a few more changes >>>>>>>>> to make, and should be able to get it up in a few days. The LP bug is >>>>>>>>> 1464387. >>>>>>>>> >>>>>>>>> Essentially, in the vpnservices table, I'm adding an IPv4 and/or >>>>>>>>> IPv6 addresses for the "local" end of VPN connections that will be >>>>>>>>> established. This is to allow alternative VPN implementation >>>>>>>>> (appliances, >>>>>>>>> separate S/W, H/W, VM based VPN, etc) to specify addresses different >>>>>>>>> than >>>>>>>>> what is available on the Neutron router. >>>>>>>>> >>>>>>>>> However, for the reference implementation, we'll use the Neutron >>>>>>>>> router's fixed_ips list (as is done today), and to handle the >>>>>>>>> migration, >>>>>>>>> I'm thinking the following is needed: >>>>>>>>> >>>>>>>>> 1) create the new columns. >>>>>>>>> 2) Identify the router for that service and obtain it's GW >>>>>>>>> fixed_ips list. >>>>>>>>> 3) Pick first IPv4 address (if any) and IPv6 address (if any), and >>>>>>>>> store in new columns. >>>>>>>>> >>>>>>>>> So I need to form a query and code to do this. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Appreciate any advise here on how to debug the migration >>>>>>>>>>> stuff... >>>>>>>>>>> >>>>>>>>>>> Paul Michali (pc_m) >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [1] >>>>>>>>>> <http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n124> >>>>>>>>>> http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/cli.py#n124 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> __________________________________________________________________________ >>>>>>>>> OpenStack Development Mailing List (not for usage questions) >>>>>>>>> Unsubscribe: >>>>>>>>> <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> >>>>>>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>>>> >>>>>>>>> >>>>>>>> __________________________________________________________________________ >>>>>>>> OpenStack Development Mailing List (not for usage questions) >>>>>>>> Unsubscribe: >>>>>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>>> >>>>>>> >>>>>> >>>>>> __________________________________________________________________________ >>>>>> OpenStack Development Mailing List (not for usage questions) >>>>>> Unsubscribe: >>>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>> >>>>>> >>>>> >>>>> __________________________________________________________________________ >>>>> OpenStack Development Mailing List (not for usage questions) >>>>> Unsubscribe: >>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>> >>>> >>>> __________________________________________________________________________ >>>> OpenStack Development Mailing List (not for usage questions) >>>> Unsubscribe: >>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>>> >>>> >>>> __________________________________________________________________________ >>>> OpenStack Development Mailing List (not for usage questions) >>>> Unsubscribe: >>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>> > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- Regards, Ann Kamyshnikova Mirantis, Inc
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev