Re: [sqlalchemy] Problem with Alembic autogenerate after updating to 1.0.2
Dear Mike, Thank you very much. Your solution worked for me. I have noticed a few minor issues as well: 1. When I try to change the type of a primary key (for example from BigInteger to Integer), if I use auto generate, the corresponding function has this key "existing_nullable=True" but does not have "nullable=True". As a result, running that function causes the MySQL error "Primary key can not be null". I always manually add "nullable=True" where it is needed. 2. When I try to drop a column which is a foreign key to another table, two actions should be taken: a. Removing foreign key constraint b. Dropping the column The sequence of these action is important. When I use Autogenerate option, usually dropping column is before removing the constraint which causes error. Again I change the sequence manually if needed. 3. When the migration code is generated and we run it, if an error comes up in midst of the migration function, it means that the code has been executed partially and the schema has been changed partially. I'm wondering if there is a solution to roll back the partial changes (similar to roll back function in sqlalchemy). The problem is that even after fixing the migration code, it is not possible to run it since it has been executed partially! I highly appreciate your effort in SQLAlchemy and Alembic and try to do a small help with improving your excellent work :) On Saturday, November 10, 2018 at 10:57:22 AM UTC+8, Mike Bayer wrote: > > OK, it's the filesystem type 2 combined with the fact that the > database is named "Test", capital T, on a case insensitive filesystem. > ill have a fix momentarily. SQLAlhcemy 1.2.12 will work for you for > now, just skip 1.2.13. > On Fri, Nov 9, 2018 at 7:57 PM > > wrote: > > > > I forgot to mention that I'm using all lowercase names for my tables and > columns. The primary keys are all "id" > > BTW I ran the command and got this result: > > > > mysql> show variables like 'lower_case_table_names'; > > ++---+ > > | Variable_name | Value | > > ++---+ > > | lower_case_table_names | 2 | > > ++---+ > > 1 row in set (0.00 sec) > > > > > > > > On Thursday, November 8, 2018 at 11:17:12 PM UTC+8, Mike Bayer wrote: > >> > >> What operating system platform are you on ? > >> > >> can you run this for me please? > >> > >> MariaDB [(none)]> show variables like 'lower_case_table_names'; > >> ++---+ > >> | Variable_name | Value | > >> ++---+ > >> | lower_case_table_names | 0 | > >> ++---+ > >> 1 row in set (0.00 sec) > >> > >> > >> On Thu, Nov 8, 2018 at 10:10 AM Mike Bayer > wrote: > >> > > >> > easier, if you can give me a "SHOW CREATE TABLE" for a table here > that > >> > has foreign key constraints and refers to a column named "id", "ID", > >> > "iD", something with those two letters in it. > >> > On Thu, Nov 8, 2018 at 9:58 AM Mike Bayer > wrote: > >> > > > >> > > On Thu, Nov 8, 2018 at 9:56 AM Mike Bayer > wrote: > >> > > > > >> > > > On Thu, Nov 8, 2018 at 3:36 AM wrote: > >> > > > > > >> > > > > Hi > >> > > > > > >> > > > > I had a problem with alembic autogenerate after updating to > version 1.0.2. I'm running MySQL 8.0.11 and Python 3.7.1 > >> > > > > No matter how my base is defined, I always get the same error. > I went through the error and found out it comes from function " > >> > > > > >> > > > Did you also update SQLAlchemy version? The check you see has to > do > >> > > > with SQLAlchemy version 1.2.13. Can you confirm that using > SQLAlchemy > >> > > > 1.2.12 resolves the issue? I can work towards making this check > more > >> > > > defensive. Additionally let me run the tests on MySQL 8 to > see if > >> > > > I'm missing something. > >> > > > >> > > no failures here, can you please provide a sample database model > that > >> > > produces this error? > >> > > > >> > > > > >> > > > > >> > > > > > >> > > > > _correct_for_mysql_bug_88718(self, fkeys, connection)" in > module "base.py" with this explaination: > >> > > > > > >> > > > > # Foreign key is always in lower case (MySQL 8.0) > >> > > > > # https://bugs.mysql.com/bug.php?id=88718 > >> > > > > # issue #4344 for SQLAlchemy > >> > > > > > >> > > > > > >> > > > > This is log of error: > >> > > > > > >> > > > > INFO [alembic.runtime.migration] Context impl MySQLImpl. > >> > > > > INFO [alembic.runtime.migration] Will assume non-transactional > DDL. > >> > > > > Traceback (most recent call last): > >> > > > > File "---/venv/bin/alembic", line 11, in > >> > > > > load_entry_point('alembic==1.0.2', 'console_scripts', > 'alembic')() > >> > > > > File > "---/venv/lib/python3.7/site-packages/alembic/config.py", line 502, in main > >> > > > >
Re: [sqlalchemy] Problem with Alembic autogenerate after updating to 1.0.2
I forgot to mention that I'm using all lowercase names for my tables and columns. The primary keys are all "id" BTW I ran the command and got this result: mysql> show variables like 'lower_case_table_names'; ++---+ | Variable_name | Value | ++---+ | lower_case_table_names | 2 | ++---+ 1 row in set (0.00 sec) On Thursday, November 8, 2018 at 11:17:12 PM UTC+8, Mike Bayer wrote: > > What operating system platform are you on ? > > can you run this for me please? > > MariaDB [(none)]> show variables like 'lower_case_table_names'; > ++---+ > | Variable_name | Value | > ++---+ > | lower_case_table_names | 0 | > ++---+ > 1 row in set (0.00 sec) > > > On Thu, Nov 8, 2018 at 10:10 AM Mike Bayer > wrote: > > > > easier, if you can give me a "SHOW CREATE TABLE" for a table here that > > has foreign key constraints and refers to a column named "id", "ID", > > "iD", something with those two letters in it. > > On Thu, Nov 8, 2018 at 9:58 AM Mike Bayer > wrote: > > > > > > On Thu, Nov 8, 2018 at 9:56 AM Mike Bayer > wrote: > > > > > > > > On Thu, Nov 8, 2018 at 3:36 AM > > wrote: > > > > > > > > > > Hi > > > > > > > > > > I had a problem with alembic autogenerate after updating to > version 1.0.2. I'm running MySQL 8.0.11 and Python 3.7.1 > > > > > No matter how my base is defined, I always get the same error. I > went through the error and found out it comes from function " > > > > > > > > Did you also update SQLAlchemy version? The check you see has to do > > > > with SQLAlchemy version 1.2.13. Can you confirm that using > SQLAlchemy > > > > 1.2.12 resolves the issue? I can work towards making this check > more > > > > defensive. Additionally let me run the tests on MySQL 8 to see > if > > > > I'm missing something. > > > > > > no failures here, can you please provide a sample database model that > > > produces this error? > > > > > > > > > > > > > > > > > > > > > _correct_for_mysql_bug_88718(self, fkeys, connection)" in module > "base.py" with this explaination: > > > > > > > > > > # Foreign key is always in lower case (MySQL 8.0) > > > > > # https://bugs.mysql.com/bug.php?id=88718 > > > > > # issue #4344 for SQLAlchemy > > > > > > > > > > > > > > > This is log of error: > > > > > > > > > > INFO [alembic.runtime.migration] Context impl MySQLImpl. > > > > > INFO [alembic.runtime.migration] Will assume non-transactional > DDL. > > > > > Traceback (most recent call last): > > > > > File "---/venv/bin/alembic", line 11, in > > > > > load_entry_point('alembic==1.0.2', 'console_scripts', > 'alembic')() > > > > > File "---/venv/lib/python3.7/site-packages/alembic/config.py", > line 502, in main > > > > > CommandLine(prog=prog).main(argv=argv) > > > > > File "---/venv/lib/python3.7/site-packages/alembic/config.py", > line 496, in main > > > > > self.run_cmd(cfg, options) > > > > > File "---/venv/lib/python3.7/site-packages/alembic/config.py", > line 479, in run_cmd > > > > > **dict((k, getattr(options, k, None)) for k in kwarg) > > > > > File "---/venv/lib/python3.7/site-packages/alembic/command.py", > line 176, in revision > > > > > script_directory.run_env() > > > > > File > "---/venv/lib/python3.7/site-packages/alembic/script/base.py", line 427, in > run_env > > > > > util.load_python_file(self.dir, 'env.py') > > > > > File > "---/venv/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 81, in > load_python_file > > > > > module = load_module_py(module_id, path) > > > > > File > "---/venv/lib/python3.7/site-packages/alembic/util/compat.py", line 82, in > load_module_py > > > > > spec.loader.exec_module(module) > > > > > File "", line 728, in > exec_module > > > > > File "", line 219, in > _call_with_frames_removed > > > > > File "alembic/env.py", line 75, in > > > > > run_migrations_online() > > > > > File "alembic/env.py", line 70, in run_migrations_online > > > > > context.run_migrations() > > > > > File "", line 8, in run_migrations > > > > > File > "---/venv/lib/python3.7/site-packages/alembic/runtime/environment.py", line > 836, in run_migrations > > > > > self.get_context().run_migrations(**kw) > > > > > File > "---/venv/lib/python3.7/site-packages/alembic/runtime/migration.py", line > 321, in run_migrations > > > > > for step in self._migrations_fn(heads, self): > > > > > File "---/venv/lib/python3.7/site-packages/alembic/command.py", > line 156, in retrieve_migrations > > > > > revision_context.run_autogenerate(rev, context) > > > > > File > "---/venv/lib/python3.7/site-packages/alembic/autogenerate/api.py", line > 415, in run_autogenerate > > > > > self._run_environment(rev, migration_context, True) > >
Re: [sqlalchemy] Problem with Alembic autogenerate after updating to 1.0.2
Dear Mike, Thank you very much for the time you put to resolve this issue. Last 2 days, I tried many times to see when I get this error. This is the procedure: 1. I created a new empty database called "Test" 2. I wrote declarative base for 3 tables 3. I used alembic autogenerate to produce migration code and then migrated to the latest version. In this stage I have 3 tables+1 alembic_version 4. I changed my base such that drop 2 existing tables and keep just one of them 5. I tried alembic autogenerate and got the error! Actually no matter my base definition is, I get the same error with alembic autogenerate unless I remove all tables from the database and build it from the scratch! Points: 1. I don't experience this error with sqlite! 2. I don't have this problem if I don't use autogenerate 3. I'm running MacOS Mojave 10.14.1 and MySQL 8.0.11 4. I'm using Python, SQLALchemy, alembic, all the latest version 5. I provide you my problematic project including base definition, alembic folder and dump of current database. 6. If you need, this is the virtual environment which I'm using: https://www.dropbox.com/s/w55iujvg0fbn16k/venv.zip?dl=0 Sorry that my virtual environment is large! I hope you can reproduce the error. Thank you again :) On Thursday, November 8, 2018 at 10:58:56 PM UTC+8, Mike Bayer wrote: > > On Thu, Nov 8, 2018 at 9:56 AM Mike Bayer > wrote: > > > > On Thu, Nov 8, 2018 at 3:36 AM > > wrote: > > > > > > Hi > > > > > > I had a problem with alembic autogenerate after updating to version > 1.0.2. I'm running MySQL 8.0.11 and Python 3.7.1 > > > No matter how my base is defined, I always get the same error. I went > through the error and found out it comes from function " > > > > Did you also update SQLAlchemy version? The check you see has to do > > with SQLAlchemy version 1.2.13. Can you confirm that using SQLAlchemy > > 1.2.12 resolves the issue? I can work towards making this check more > > defensive. Additionally let me run the tests on MySQL 8 to see if > > I'm missing something. > > no failures here, can you please provide a sample database model that > produces this error? > > > > > > > > > > > _correct_for_mysql_bug_88718(self, fkeys, connection)" in module > "base.py" with this explaination: > > > > > > # Foreign key is always in lower case (MySQL 8.0) > > > # https://bugs.mysql.com/bug.php?id=88718 > > > # issue #4344 for SQLAlchemy > > > > > > > > > This is log of error: > > > > > > INFO [alembic.runtime.migration] Context impl MySQLImpl. > > > INFO [alembic.runtime.migration] Will assume non-transactional DDL. > > > Traceback (most recent call last): > > > File "---/venv/bin/alembic", line 11, in > > > load_entry_point('alembic==1.0.2', 'console_scripts', 'alembic')() > > > File "---/venv/lib/python3.7/site-packages/alembic/config.py", line > 502, in main > > > CommandLine(prog=prog).main(argv=argv) > > > File "---/venv/lib/python3.7/site-packages/alembic/config.py", line > 496, in main > > > self.run_cmd(cfg, options) > > > File "---/venv/lib/python3.7/site-packages/alembic/config.py", line > 479, in run_cmd > > > **dict((k, getattr(options, k, None)) for k in kwarg) > > > File "---/venv/lib/python3.7/site-packages/alembic/command.py", line > 176, in revision > > > script_directory.run_env() > > > File "---/venv/lib/python3.7/site-packages/alembic/script/base.py", > line 427, in run_env > > > util.load_python_file(self.dir, 'env.py') > > > File "---/venv/lib/python3.7/site-packages/alembic/util/pyfiles.py", > line 81, in load_python_file > > > module = load_module_py(module_id, path) > > > File "---/venv/lib/python3.7/site-packages/alembic/util/compat.py", > line 82, in load_module_py > > > spec.loader.exec_module(module) > > > File "", line 728, in > exec_module > > > File "", line 219, in > _call_with_frames_removed > > > File "alembic/env.py", line 75, in > > > run_migrations_online() > > > File "alembic/env.py", line 70, in run_migrations_online > > > context.run_migrations() > > > File "", line 8, in run_migrations > > > File > "---/venv/lib/python3.7/site-packages/alembic/runtime/environment.py", line > 836, in run_migrations > > > self.get_context().run_migrations(**kw) > > > File > "---/venv/lib/python3.7/site-packages/alembic/runtime/migration.py", line > 321, in run_migrations > > > for step in self._migrations_fn(heads, self): > > > File "---/venv/lib/python3.7/site-packages/alembic/command.py", line > 156, in retrieve_migrations > > > revision_context.run_autogenerate(rev, context) > > > File > "---/venv/lib/python3.7/site-packages/alembic/autogenerate/api.py", line > 415, in run_autogenerate > > > self._run_environment(rev, migration_context, True) > > > File > "---/venv/lib/python3.7/site-packages/alembic/autogenerate/api.py", line > 451, in _run_environment > > >
[sqlalchemy] Problem with Alembic autogenerate after updating to 1.0.2
Hi I had a problem with alembic autogenerate after updating to version 1.0.2. I'm running MySQL 8.0.11 and Python 3.7.1 No matter how my base is defined, I always get the same error. I went through the error and found out it comes from function " _correct_for_mysql_bug_88718(self, fkeys, connection)" in module "base.py" with this explaination: # Foreign key is always in lower case (MySQL 8.0) # https://bugs.mysql.com/bug.php?id=88718 # issue #4344 for SQLAlchemy This is log of error: INFO [alembic.runtime.migration] Context impl MySQLImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. Traceback (most recent call last): File "---/venv/bin/alembic", line 11, in load_entry_point('alembic==1.0.2', 'console_scripts', 'alembic')() File "---/venv/lib/python3.7/site-packages/alembic/config.py", line 502, in main CommandLine(prog=prog).main(argv=argv) File "---/venv/lib/python3.7/site-packages/alembic/config.py", line 496, in main self.run_cmd(cfg, options) File "---/venv/lib/python3.7/site-packages/alembic/config.py", line 479, in run_cmd **dict((k, getattr(options, k, None)) for k in kwarg) File "---/venv/lib/python3.7/site-packages/alembic/command.py", line 176, in revision script_directory.run_env() File "---/venv/lib/python3.7/site-packages/alembic/script/base.py", line 427, in run_env util.load_python_file(self.dir, 'env.py') File "---/venv/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 81, in load_python_file module = load_module_py(module_id, path) File "---/venv/lib/python3.7/site-packages/alembic/util/compat.py", line 82, in load_module_py spec.loader.exec_module(module) File "", line 728, in exec_module File "", line 219, in _call_with_frames_removed File "alembic/env.py", line 75, in run_migrations_online() File "alembic/env.py", line 70, in run_migrations_online context.run_migrations() File "", line 8, in run_migrations File "---/venv/lib/python3.7/site-packages/alembic/runtime/environment.py", line 836, in run_migrations self.get_context().run_migrations(**kw) File "---/venv/lib/python3.7/site-packages/alembic/runtime/migration.py", line 321, in run_migrations for step in self._migrations_fn(heads, self): File "---/venv/lib/python3.7/site-packages/alembic/command.py", line 156, in retrieve_migrations revision_context.run_autogenerate(rev, context) File "---/venv/lib/python3.7/site-packages/alembic/autogenerate/api.py", line 415, in run_autogenerate self._run_environment(rev, migration_context, True) File "---/venv/lib/python3.7/site-packages/alembic/autogenerate/api.py", line 451, in _run_environment autogen_context, migration_script) File "---/venv/lib/python3.7/site-packages/alembic/autogenerate/compare.py", line 22, in _populate_migration_script _produce_net_changes(autogen_context, upgrade_ops) File "---/venv/lib/python3.7/site-packages/alembic/autogenerate/compare.py", line 48, in _produce_net_changes autogen_context, upgrade_ops, schemas File "---/venv/lib/python3.7/site-packages/alembic/util/langhelpers.py", line 313, in go fn(*arg, **kw) File "---/venv/lib/python3.7/site-packages/alembic/autogenerate/compare.py", line 75, in _autogen_for_tables inspector, upgrade_ops, autogen_context) File "---/venv/lib/python3.7/site-packages/alembic/autogenerate/compare.py", line 137, in _compare_tables inspector.reflecttable(t, None) File "---/venv/lib/python3.7/site-packages/sqlalchemy/engine/reflection.py", line 633, in reflecttable exclude_columns, _extend_on, reflection_options) File "---/venv/lib/python3.7/site-packages/sqlalchemy/engine/reflection.py", line 729, in _reflect_fk table_name, schema, **table.dialect_kwargs) File "---/venv/lib/python3.7/site-packages/sqlalchemy/engine/reflection.py", line 447, in get_foreign_keys **kw) File "", line 2, in get_foreign_keys File "---/venv/lib/python3.7/site-packages/sqlalchemy/engine/reflection.py", line 54, in cache ret = fn(self, con, *args, **kw) File "---/venv/lib/python3.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2081, in get_foreign_keys self._correct_for_mysql_bug_88718(fkeys, connection) File "---/venv/lib/python3.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2126, in _correct_for_mysql_bug_88718 for col in fkey['referred_columns'] File "---/venv/lib/python3.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2126, in for col in fkey['referred_columns'] KeyError: 'id' -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from
[sqlalchemy] How to implement a multi-level single table polymorphism using sqlalchemy?
Hi all I have a phone table in MySQL which has morphology based on its two columns, "is_deleted" and "is_active" If "is_deleted" column is 1, then we don't care about "is_active" column. If "is_deleted" column is 0, it means that the phone exists and can be either active or inactive based on "is_active" column What I have read, looks like this: class FullPhone(Base): __tablename__ = 'phone' is_deleted = Column(TINYINT, nullable=False, server_default='0') __mapper_args__ = { 'polymorphic_identity': 1, 'polymorphic_on': is_deleted } class ExistingPhone(FullPhone): is_active = Column(TINYINT, server_default='1') __mapper_args__ = { 'polymorphic_identity': 0, 'polymorphic_on': is_active } class ActivePhone(ExistingPhone): __mapper_args__ = { 'polymorphic_identity': 1, } class InactivePhone(ExistingPhone): __mapper_args__ = { 'polymorphic_identity': 0, } When I run this code, I got a warning: Reassigning polymorphic association for identity 0 from to : Check for duplicate use of 0 as value for polymorphic_identity. It seems that: the polymorphic identity '0' of "ExistingPhone" which is a condition on "is_deleted" column is mistaken by: the polymorphic identity '0' of "InactivePhone" which is another condition on "is_active" column I appreciate if someone can help me with this structure. Thank you. -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.