#36754: Bug in GeneratedField when it references a related field (e.g.
ForeignKey)
with 2 conditions (django happens to create multiple 000x_inital.py for the
app && the ForeignKey is first initialized in the later file
000x_inital.py)
-------------------------------------+-------------------------------------
Reporter: Ou7law007 | Type: Bug
Status: new | Component: Database
| layer (models, ORM)
Version: 5.2 | Severity: Normal
Keywords: migrations | Triage Stage:
autodetector | Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
When the autodetector builds CreateModel operations, it defers related
fields (FK/M2M) into separate AddField operations. However, GeneratedField
expressions are evaluated before those deferred fields exist in the
CreateModel operation, causing the generated expression to reference a
non‑existent column name. This results in inconsistent or invalid initial
migrations on new apps.
== This is not easy to reproduce
In a small Django project, the dependency graph between apps is small.
When Django detects initial migrations, it produces a single 0001_initial
file per app. And since all fields (including FK targets used inside
GeneratedField expressions) are created in that one file, Django never
attempts to evaluate a GeneratedField expression that references a
relationship **that does not yet exist**.
== When the bug happens
In larger projects, Django may split initial migrations into multiple
files (0001_initial, 0002_initial, sometimes 0003_initial, etc.).
When this happens:
- Django may place a GeneratedField into 0001_initial.
- Django may place the ForeignKey needed by the GeneratedField expression
into 0002_initial.
- Django then tries to serialize the GeneratedField expression in
0001_initial, but the FK field it references does not yet exist.
- This results in the following error:
{{{django.core.exceptions.FieldError: Cannot resolve keyword 'category_id'
into field. Choices are: id, ... etc.}}} with {{{category_id}}} being a
foreign key to a model that hasn't been processed yet.
In other words, the bug only appears when Django generates an initial
migration before creating the FK that a GeneratedField depends on.
Also, note that it only happens on **INITIAL** migrations, which makes it
even harder to reproduce, because you need an existing large project that
needs to initialize all its models' migrations for the first time.
== Example (if you copy paste into a small project, you **won't** be able
to reproduce the issue, read above for why)
app_a:
{{{
class Category(models.Model):
name = models.CharField(max_length=100)
}}}
app_b:
{{{
class Item(models.Model):
category = models.ForeignKey("A.Category", on_delete=models.CASCADE)
# Problematic GeneratedField referring to category_id
somefield = models.GeneratedField(
expression=Concat(
F("category_id"), # or just category_id, makes no difference,
both are bugged
Value("-"),
F("id"),
),
output_field=models.CharField(max_length=50),
db_persist=True,
unique=True,
)
}}}
{{{
B/0001_initial.py # creates Item without the category FK
B/0002_initial.py # adds the category FK
}}}
== Workarounds
1. Manually merge migrations i.e. move the FK field (or other
dependencies) from 0002_initial.py into 0001_initial.py to **ensure that
all fields used by GeneratedField expressions are created together**.
2. Improt the user model inside the initial custom user migration!!!!
(That's a werid one) This one is based on django cookiecutter which
creates a {{{users}}} app with a custom user model. I noticed that
importing that custom model inside the initial user migration file i.e.
put {{{import myproject.users.models}}} inside
{{{myproject/users/migrations/0001_initial.py}}}, which is the default for
django-cookiecutter projects, that's why if you're using cookiecutter, you
will not have this bug unless you delete all migration files after your
project grows enough and then try to migrate from scratch, then you notice
the only difference between the old and new initial migration files is
this line {{{import myproject.users.models}}} which fixes the bug. I also
noticed that other apps had up to 3 000x_initial.py files but once I added
{{{import myproject.users.models}}}, they went down to just one for each
app.
Briefly and as a summary, the issue is that if a generated field
(`somefield` in the example above) has a reference to a foreign key (or
any other related field) (`category_id` in the example above) **AND**
django happens to create multiple 000x_initial.py migration files for that
app (see 1) **AND** the foreign relation field is declared in a later file
than the generated field, then the bug happens because django processes
the generated field, but the field that is mentioned inside of it.
I'd also appreciate it, if someone can answer this question: Is it an
expected behavior that importing {{{import myproject.users.models}}} i.e.
the custom user model in the users' apps initial migration file i.e. in
{{{myproject/userss/migrations/0001_initial.py}}} causes less initial
migration files to be generated for each app? Is this normal? Because this
is what seems to solve the issue (or make it not even an issue)
1.
https://docs.djangoproject.com/en/5.2/topics/migrations/#:~:text=but%20in%20some%20cases%20of%20complex%20model%20interdependencies%20it%20may%20have%20two%20or%20more
--
Ticket URL: <https://code.djangoproject.com/ticket/36754>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/django-updates/0107019ab6909ffd-d37a37f3-f748-4f3a-a937-9e4570687fe3-000000%40eu-central-1.amazonses.com.
