#32302: Permit migrations in non-namespace packages that don't have __file__
-------------------------------------+-------------------------------------
Reporter: William Schwartz | Owner: William
Type: | Schwartz
Cleanup/optimization | Status: closed
Component: Migrations | Version: master
Severity: Normal | Resolution: fixed
Keywords: migrations freezers | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by William Schwartz):
== Appendix: Research Notes
'''''This ticket is closed. You need read no further unless you're
researching a different issue related to the Python import system or
Django's migration loading.'''''
While doing my research this ticket, I took a lot of notes that didn't
make it into the actual report. Here they are, in case it helps someone
else down the line.
=== More background on `__file__`
`__file__` is set by the import machinery from
[https://docs.python.org/3.9/library/importlib.html#importlib.machinery.ModuleSpec.origin
importlib.machinery.ModuleSpec.origin], which may be `None`:
> Name of the place from which the module is loaded, e.g. “builtin” for
built-in modules and the filename for modules loaded from source. Normally
“origin” should be set, but it may be None (the default) which indicates
it is unspecified (e.g. for namespace packages).
One frozen Python environment, PyOxidizer,
[https://pyoxidizer.readthedocs.io/en/stable/oxidized_importer_behavior_and_compliance.html
#file-and-cached-module-attributes relies] on this API rule.
=== Proving a module is a namespace package
[https://docs.python.org/3.9/whatsnew/3.7.html#other-cpython-
implementation-changes Since Python 3.7] fixed
[https://bugs.python.org/issue32303 bpo-32303], it is no longer possible
to prove definitively that a module is a namespace package. In Python 3.6
and earlier, a module's
[https://docs.python.org/3.9/reference/import.html#__loader__ __loader__]
was `None` if and only if the module was a namespace package. This was
because [https://www.python.org/dev/peps/pep-0451/#namespace-packages PEP
451] required that a [https://docs.python.org/3.9/glossary.html#term-
finder finder]'s
[https://docs.python.org/3.9/library/importlib.html#importlib.abc.MetaPathFinder.find_spec
find_spec] set
[https://docs.python.org/3.9/library/importlib.html#importlib.machinery.ModuleSpec.loader
loader] to `None`:
> Currently a path entry finder may return (None, portions) from
[https://docs.python.org/3.9/library/importlib.html#importlib.abc.PathEntryFinder.find_loader
find_loader()] to indicate it found part of a possible namespace package.
To achieve the same effect, find_spec() must return a spec with "loader"
set to None (a.k.a. not set) and with submodule_search_locations set....
In Python 3.9, calling
[https://docs.python.org/3.9/library/importlib.html#importlib.util.find_spec
importlib.util.find_spec] on the name of a not-yet-imported namespace
package returned a spec whose `loader` is `None`, but once the module is
imported, the returned spec's `loader` is an instance of
`_frozen_importlib_external._NamespaceLoader`. Note that
`_NamespaceLoader` is an implementation detail of CPython.
We must resist the temptation to rely on Python's
[https://docs.python.org/3.9/reference/import.html#loading documented code
for detecting namespace packages]:
> {{{#!python
> if spec.origin is None and spec.submodule_search_locations is not None:
> # namespace package
> sys.modules[spec.name] = module
> }}}
This documentation is patently falsified by a cursory review of Python's
[https://github.com/python/cpython/blob/4140f10a16f06c32fd49f9e21fb2a53abe7357f0/Lib/importlib/_bootstrap.py#L511-L538
source code], which clearly relies on `loader` being `None`.
See also [https://www.python.org/dev/peps/pep-0420/#differences-between-
namespace-packages-and-regular-packages PEP-420's summary of differences
between namespace and regular packages], which is now somewhat outdated.
=== History of the migration loader's no-namespace check
The reason, given both in the
[https://github.com/django/django/blob/2a76f4313423a3b91caade4fce71790630ef9152/django/db/migrations/loader.py#L91-L95
source code comments] and in [https://groups.google.com/g/django-
developers/c/GVHMH2ciAnk/m/vbIPbZuSBQAJ discussions among maintainers],
for the migration loader's check that an app's `migrations` package is not
a namespace package is to discourage the latter's use.
The current namespace-check implementation was originally introduced in
[https://github.com/django/django/pull/1541 GH-1541], fixing #21015, to
skip empty directories. [https://github.com/django/django/pull/9665
GH-9665] allowed for the possibility that `__file__` be `None` rather than
missing on namespace packages because of
[https://bugs.python.org/issue32305 bpo-32305], which was
[https://docs.python.org/3.9/whatsnew/3.7.html#other-cpython-
implementation-changes fixed in Python 3.7]. The implementation was
briefly removed in [https://github.com/django/django/pull/11141 GH-11141]
to fix #30300, but revived in [https://github.com/django/django/pull/13218
GH-13218] [https://groups.google.com/g/django-
developers/c/GVHMH2ciAnk/m/vbIPbZuSBQAJ because]
> Supporting namespace packages without a real use case seems contrary to
the consensus in this thread (which I see as not promoting implicit
namespace packages). Based on that consensus, my inclination wouldn't be
to try to make Django work with as few `__init__.py` files as possible.
--
Ticket URL: <https://code.djangoproject.com/ticket/32302#comment:5>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/068.b9caa78ccf0eb6490acd9a80ad6dc867%40djangoproject.com.
