#29632: After migrating a Django project from Python 2 to 3, db migration of
django.contrib.sites fails with "TypeError: attribute name must be string,
not 'bytes'"
-------------------------------------+-------------------------------------
Reporter: Florian | Owner: nobody
Mayer |
Type: Bug | Status: new
Component: | Version: master
Migrations | Keywords: migration, unicode,
Severity: Normal | bytestring, TypeError
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 1
UI/UX: 0 |
-------------------------------------+-------------------------------------
I'm migrating an existing Django project from Python 2.7 / Django 1.11 to
Python 3.6 / Django 2.0.7 (latest stable at the time of writing).
Migrations are finicky as they (used to) be created without `from
__future__ import unicode_literals`, therefore all strings (such as model
manager names e.g. "'objects'") would be interpreted as bytestrings in
Python 3.x.
[https://code.djangoproject.com/ticket/24701 Ticket 24701] is the closest
issue I could find relating to this error, and it led to Django now
inserting the `unicode_literals` import into newly created migrations.
This however doesn't fix existing migrations, and while Django's
migrations are doing now a decent job to write plain un-prefixed
`"strings"` in field names, sometimes a model manager name or a more
exotic migration command's argument slips through as `b"explicitly
prefixed bytestring"`.
== Environment ==
* [https://github.com/dbca-wa/wastd/blob/master/.circleci/config.yml
CircleCI v2 build] using Python 3.6.1 (freshly upgraded from Python 2.7)
* [https://github.com/dbca-wa/wastd/blob/master/requirements/base.txt
requirements] Django 2.0.7
* The Django project was written Django 1.9.7 to 1.11.x, and recently
upgraded to Django 2.x.
* The migrations were created using Django 1.9.7 to 1.11.x
* Third party app migrations are potentially ancient
== Stack trace ==
[https://circleci.com/gh/dbca-wa/wastd/823 Failing CircleCI build] fails
at `./manage.py migrate`.
Can reproduce on Ubuntu 16.04 (Python 3.6.x Anaconda) and 14.04 (Python
3.4.3).
{{{
Running migrations:
Applying contenttypes.0001_initial... OK
Applying contenttypes.0002_remove_content_type_name... OK
[... working migrations ...]
Applying sites.0001_initial...Traceback (most recent call last):
File "manage.py", line 12, in <module>
execute_from_command_line(sys.argv)
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/core/management/__init__.py", line 371, in
execute_from_command_line
utility.execute()
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/core/management/__init__.py", line 365, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/core/management/commands/test.py", line 26, in
run_from_argv
super().run_from_argv(argv)
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/core/management/base.py", line 335, in execute
output = self.handle(*args, **options)
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/core/management/commands/test.py", line 59, in handle
failures = test_runner.run_tests(test_labels)
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/test/runner.py", line 601, in run_tests
old_config = self.setup_databases()
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/test/runner.py", line 548, in setup_databases
self.parallel, **kwargs
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/test/utils.py", line 176, in setup_databases
serialize=connection.settings_dict.get('TEST', {}).get('SERIALIZE',
True),
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/db/backends/base/creation.py", line 68, in create_test_db
run_syncdb=True,
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/core/management/__init__.py", line 141, in call_command
return command.execute(*args, **defaults)
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/core/management/base.py", line 335, in execute
output = self.handle(*args, **options)
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/core/management/commands/migrate.py", line 200, in handle
fake_initial=fake_initial,
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/db/migrations/executor.py", line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake,
fake_initial=fake_initial)
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/db/migrations/executor.py", line 147, in
_migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake,
fake_initial=fake_initial)
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/db/migrations/executor.py", line 244, in apply_migration
state = migration.apply(state, schema_editor)
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/db/migrations/migration.py", line 112, in apply
operation.state_forwards(self.app_label, project_state)
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/db/migrations/operations/models.py", line 86, in
state_forwards
list(self.managers),
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/db/migrations/state.py", line 97, in add_model
self.reload_model(app_label, model_name)
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/db/migrations/state.py", line 158, in reload_model
self._reload(related_models)
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/db/migrations/state.py", line 191, in _reload
self.apps.render_multiple(states_to_be_rendered)
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/db/migrations/state.py", line 306, in render_multiple
model.render(self)
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/db/migrations/state.py", line 575, in render
return type(self.name, bases, body)
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/db/models/base.py", line 152, in __new__
new_class.add_to_class(obj_name, obj)
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/db/models/base.py", line 315, in add_to_class
value.contribute_to_class(cls, name)
File "/home/circleci/repo/venv/lib/python3.6/site-
packages/django/db/models/manager.py", line 115, in contribute_to_class
setattr(model, name, ManagerDescriptor(self))
TypeError: attribute name must be string, not 'bytes'
}}}
tl;dr: The `name`, a value coming from the migration `sites.0001_initial`,
was interpreted by `setattr(model, name, ManagerDescriptor(self))` as a
bytestring.
== Steps in detail ==
* django.contrib.sites
[https://github.com/django/django/blob/master/django/contrib/sites/migrations/0001_initial.py
migration 0001] is missing the `unicode_literals` statement. Under Python
3, e.g. its
[https://github.com/django/django/blob/master/django/contrib/sites/migrations/0001_initial.py#L28
model manager name] `'objects'` will be interpreted as bytestring
`b'objects'`.
* While running `./manage.py migrate`, the
[https://github.com/django/django/blob/master/django/db/models/manager.py#L113
django.db.models.manager L113] reads the migration's model manager name
expecting a unicode string, and not defending against a bytestring.
I was able to fix my own existing migrations (created in Django 1.x)
either by hand (inserting the `unicode_literals` import, or deleting the
occasional explicit `b""` prefix), or by squashing multiple migrations
(where too numerous to manually edit) under Django 2.x, which inserted the
`unicode_literals` statement into my newly created migrations.
However, this approach won't work with third party migrations without
manually patching them (which won't work on e.g. CI and is brittle) or
forking (creating a maintenance trail for me).
== Suggested patches ==
* Primarily, add `from __future__ import unicode_literals` to
django.contrib.sites'
[https://github.com/django/django/blob/master/django/contrib/sites/migrations/0001_initial.py
migration 0001].
* As a longer term fix, replace
[https://github.com/django/django/blob/master/django/db/models/manager.py#L113
django.db.models.manager L113] `setattr(model, name,
ManagerDescriptor(self))` with `setattr(model, force_text(name),
ManagerDescriptor(self))` (and import `force_text`).
* If a bytestring is found, a Django warning should prompt the user to
clean up the offending migration by adding the `unicode_literals` import.
This would be more graceful and efficient than presenting the TypeError.
The benefit of patching db.models.manager would be that all Python
2.x-style migrations, especially of third party packages, would work in
Python 3.x-based Django projects without the need to patch them.
Has anyone else encountered this bug? Happy to PR whichever fix is deemed
most appropriate (bear with me, first Django bug report).
--
Ticket URL: <https://code.djangoproject.com/ticket/29632>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/051.4f6ad3106d4c2330e58f39aabe7927e9%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
