Re: [Python-Dev] importlib hooray!

[Brett Cannon]

On Wed, Feb 6, 2013 at 4:26 AM, Thomas Heller  wrote:

> I have become a fan of the new python 3.3 importlib
> in the last few days.
>

Glad it's working out for you!


>
> It has allowed me to write a ModuleMapper which I put into
> sys.metapath (in sitecustomize.py, for Python 3.3).
>
> This mapper currently does rename modules like 'Queue' or '_winreg'
> to the Python3 modules 'queue' or 'winreg' without the need to change
> my 2.7 sourcecode (changing the sourcecode is required when using
> six.moves to write 2.x/3.x compatible code).
>
> The six.moves approach has another problem with freeze tools (py2exe
> for example):  ModuleFinder does not find the moved modules because
> it cannot track __import__().
>
>
> I have also started a new modulefinder which uses importlib to find
> modules; this was quite easy since I could copy a lot of code from
> importlib._bootstrap.  The great thing is that this new modulefinder
> is much better than the old one:  It finds modules in zipped eggs
> or other zip-archives which are on sys.path; also it finds the renamed
> modules correctly that my ModuleMapper has mapped.
>
>
> The only thing that I am missing is that it is a bit of guesswork to
> find out the type of the module that importlib.find_loader() has found.
>
> Previously imp.find_module returned a tuple containing the type of
> module found (like imp.PY_SOURCE), I have to poke around in some
> attributes of the result of importlib.find_loader() to guess the type.
>

You should be able to tell with an isinstance check on the returned loader
and not have to look at any attributes (e.g. source will subclass
SourceLoader, frozen will subclass FrozenImporter, etc.). If it continues
to be an issue just let me know and we can see what we can do (this is
obviously a niche need and so I would prefer not to change the loader API
to explicitly accommodate this, but it can be done).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] importlib hooray!

[Thomas Heller]

I have become a fan of the new python 3.3 importlib
in the last few days.

It has allowed me to write a ModuleMapper which I put into
sys.metapath (in sitecustomize.py, for Python 3.3).

This mapper currently does rename modules like 'Queue' or '_winreg'
to the Python3 modules 'queue' or 'winreg' without the need to change
my 2.7 sourcecode (changing the sourcecode is required when using
six.moves to write 2.x/3.x compatible code).

The six.moves approach has another problem with freeze tools (py2exe
for example):  ModuleFinder does not find the moved modules because
it cannot track __import__().


I have also started a new modulefinder which uses importlib to find
modules; this was quite easy since I could copy a lot of code from
importlib._bootstrap.  The great thing is that this new modulefinder
is much better than the old one:  It finds modules in zipped eggs
or other zip-archives which are on sys.path; also it finds the renamed
modules correctly that my ModuleMapper has mapped.


The only thing that I am missing is that it is a bit of guesswork to
find out the type of the module that importlib.find_loader() has found.

Previously imp.find_module returned a tuple containing the type of
module found (like imp.PY_SOURCE), I have to poke around in some
attributes of the result of importlib.find_loader() to guess the type.

Thomas

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Eric V. Smith]

On 04/20/2012 10:59 AM, Brett Cannon wrote:
> 
> 
> On Fri, Apr 20, 2012 at 09:54, Eric V. Smith  > wrote:
> 
> On 04/14/2012 02:12 PM, Brett Cannon wrote:
> > My multi-year project -- started in 2006 according to my blog -- to
> > rewrite import in pure Python and then bootstrap it into CPython as
> > *the* implementation of __import__() is finally over (mostly)!
> 
> Maybe I'm missing something, but it seems that I need to run
> importlib._bootstrap._install(sys, _imp) manually in order to make
> __import__ be importlib's version. Is that not supposed to happen
> automatically?
> 
> 
> It's happening automatically. If you look in Python/import.c you will
> notice that the code that __import__() eventually calls is calling out
> into the Python code. There is still some C code in order to accelerate
> the case of hitting sys.modules.

Okay. But I'm running make, and that's succeeding (and it looks like it
does the right thing), yet it doesn't appear to be picking up my changes
to _bootstrap.py automatically. I'll keep investigating.

Eric.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Brett Cannon]

On Fri, Apr 20, 2012 at 11:02, Eric Snow wrote:

> On Fri, Apr 20, 2012 at 7:54 AM, Eric V. Smith  wrote:
> > On 04/14/2012 02:12 PM, Brett Cannon wrote:
> >> My multi-year project -- started in 2006 according to my blog -- to
> >> rewrite import in pure Python and then bootstrap it into CPython as
> >> *the* implementation of __import__() is finally over (mostly)!
> >
> > Maybe I'm missing something, but it seems that I need to run
> > importlib._bootstrap._install(sys, _imp) manually in order to make
> > __import__ be importlib's version. Is that not supposed to happen
> > automatically?
>
> In the default tip (3.3a2+), importlib.__import__ is already
> bootstrapped, so you don't need mess with anything.  As well, in any
> of the 3.x versions you can bind builtins.__import__ to
> importlib.__import__.
>
> If you are making changes to importlib (essentially, changes in
> Lib/importlib/_bootstrap.py), you must re-build (make) cpython in
> order for your changes to get pulled into the frozen copy of
> importlib.  Until you do that, the built-in import machinery will be
> the one that existed before your changes.  You could also re-bind
> builtins.__import__ to try out the changes without having to re-build,
> but ultimately your changes will have to get frozen (into
> Python/importlib.h) and will be part of the commit of your changes to
> importlib.
>
> Likely you already know all this, but just in case...  :)


And if you want to run a test using importlib instead of the frozen code
you can use importlib.test.regrtest to handle the injection for you.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Eric Snow]

On Fri, Apr 20, 2012 at 7:54 AM, Eric V. Smith  wrote:
> On 04/14/2012 02:12 PM, Brett Cannon wrote:
>> My multi-year project -- started in 2006 according to my blog -- to
>> rewrite import in pure Python and then bootstrap it into CPython as
>> *the* implementation of __import__() is finally over (mostly)!
>
> Maybe I'm missing something, but it seems that I need to run
> importlib._bootstrap._install(sys, _imp) manually in order to make
> __import__ be importlib's version. Is that not supposed to happen
> automatically?

In the default tip (3.3a2+), importlib.__import__ is already
bootstrapped, so you don't need mess with anything.  As well, in any
of the 3.x versions you can bind builtins.__import__ to
importlib.__import__.

If you are making changes to importlib (essentially, changes in
Lib/importlib/_bootstrap.py), you must re-build (make) cpython in
order for your changes to get pulled into the frozen copy of
importlib.  Until you do that, the built-in import machinery will be
the one that existed before your changes.  You could also re-bind
builtins.__import__ to try out the changes without having to re-build,
but ultimately your changes will have to get frozen (into
Python/importlib.h) and will be part of the commit of your changes to
importlib.

Likely you already know all this, but just in case...  :)

-eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Brett Cannon]

On Fri, Apr 20, 2012 at 09:54, Eric V. Smith  wrote:

> On 04/14/2012 02:12 PM, Brett Cannon wrote:
> > My multi-year project -- started in 2006 according to my blog -- to
> > rewrite import in pure Python and then bootstrap it into CPython as
> > *the* implementation of __import__() is finally over (mostly)!
>
> Maybe I'm missing something, but it seems that I need to run
> importlib._bootstrap._install(sys, _imp) manually in order to make
> __import__ be importlib's version. Is that not supposed to happen
> automatically?


It's happening automatically. If you look in Python/import.c you will
notice that the code that __import__() eventually calls is calling out into
the Python code. There is still some C code in order to accelerate the case
of hitting sys.modules.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Eric V. Smith]

On 04/14/2012 02:12 PM, Brett Cannon wrote:
> My multi-year project -- started in 2006 according to my blog -- to
> rewrite import in pure Python and then bootstrap it into CPython as
> *the* implementation of __import__() is finally over (mostly)!

Maybe I'm missing something, but it seems that I need to run
importlib._bootstrap._install(sys, _imp) manually in order to make
__import__ be importlib's version. Is that not supposed to happen
automatically?

Eric.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Brett Cannon]

On Tue, Apr 17, 2012 at 13:39, Antoine Pitrou  wrote:

> On Tue, 17 Apr 2012 11:41:32 -0400
> Brett Cannon  wrote:
> >
> > Actually Cython would help with a subtle maintenance burden of
> maintaining
> > *any* C code for import. Right now,
> > Python/import.c:PyImport_ImportModuleLevelObject() is an accelerated C
> > version of importlib.__import__() through checking sys.modules, after
> which
> > it calls into the Python code. Cython would do away with that C
> > acceleration code (which I have already had to modify once and Antoine
> > found a couple refleaks in).
>
> Would it? That's assuming Cython would be smart enough to do the
> required optimizations.
>

Yes, it is an assumption I'm making. I also assume we wouldn't make a
change like this w/o taking the time to run importlib through Cython and
seeing how the performance numbers come out.

-Brett


>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Antoine Pitrou]

On Tue, 17 Apr 2012 11:41:32 -0400
Brett Cannon  wrote:
> 
> Actually Cython would help with a subtle maintenance burden of maintaining
> *any* C code for import. Right now,
> Python/import.c:PyImport_ImportModuleLevelObject() is an accelerated C
> version of importlib.__import__() through checking sys.modules, after which
> it calls into the Python code. Cython would do away with that C
> acceleration code (which I have already had to modify once and Antoine
> found a couple refleaks in).

Would it? That's assuming Cython would be smart enough to do the
required optimizations.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Brett Cannon]

On Tue, Apr 17, 2012 at 06:43, Eric V. Smith  wrote:

> On 4/17/2012 5:52 AM, Antoine Pitrou wrote:
> > On Mon, 16 Apr 2012 20:41:56 -0400
> > Brett Cannon  wrote:
> >> On Mon, Apr 16, 2012 at 20:27, Antoine Pitrou 
> wrote:
> >>
> >>> On Tue, 17 Apr 2012 01:11:14 +0200
> >>> Georg Brandl  wrote:
> 
>  No, it's not just an existing Python, it is (at least currently) the
> same
>  version of Python being built.  Therefore I wrote about the
> bootstrapping
>  problems when bytecode changes.
> 
>  Depending on Cython is better in that it breaks the bootstrapping
> cycle,
>  but on the other hand the C code may need to be regenerated when the C
> >>> API
>  changes in an incompatible way.
> >>>
> >>> Cython OTOH probably needs Python 2.x, which isn't that great for
> >>> building Python 3. And requiring Cython for developing is not very
> >>> contributor-friendly.
> >>>
> >>
> >> Well, required to regenerate _frozen_importlib, but nothing else. I mean
> >> making fixes go into importlib directly and get tested that way, not
> >> through __import__(). So really Cython would only be needed when
> >> importlib._bootstrap has been changed and you are making a commit.
> >
> > That's still a large dependency to bring in, while we already have a
> > working solution.
> > I'd understand using Cython to develop some new extension module which
> > requires linking against a C library (and is thus impossible to write
> > in pure Python). But for importlib that's totally non-necessary.
> >
> > I guess I'm -1 on it.
>
> I agree. If the problem we're trying to solve is that the generated file
> isn't always rebuilt, bringing in a large dependency like Cython seems
> like overkill to me.
>

Actually Cython would help with a subtle maintenance burden of maintaining
*any* C code for import. Right now,
Python/import.c:PyImport_ImportModuleLevelObject() is an accelerated C
version of importlib.__import__() through checking sys.modules, after which
it calls into the Python code. Cython would do away with that C
acceleration code (which I have already had to modify once and Antoine
found a couple refleaks in).


>
> We basically have a working solution now (thanks, Brett). I think we
> should focus on getting it polished. Maybe we can bring in Cython in a
> later release, if in the 3.4 timeframe it still seems like we have a
> problem to solve. I suspect things will be working fine.


I don't view this discussion as work/not work but more of work/work better
(possibly; I have severe bias here to cut the C code down to zilch since I
don't want to write anymore of it so I'm definitely not going to make any
final call on this topic).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[R. David Murray]

On Tue, 17 Apr 2012 01:11:14 +0200, Georg Brandl  wrote:
> On 16.04.2012 18:15, R. David Murray wrote:
> > I don't see how depending on Cython is better than depending on having
> > an existing Python.
> 
> No, it's not just an existing Python, it is (at least currently) the same
> version of Python being built.  Therefore I wrote about the bootstrapping
> problems when bytecode changes.

Ah, yes, I had missed that subtlety.

--David
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Eric V. Smith]

On 4/17/2012 5:52 AM, Antoine Pitrou wrote:
> On Mon, 16 Apr 2012 20:41:56 -0400
> Brett Cannon  wrote:
>> On Mon, Apr 16, 2012 at 20:27, Antoine Pitrou  wrote:
>>
>>> On Tue, 17 Apr 2012 01:11:14 +0200
>>> Georg Brandl  wrote:

 No, it's not just an existing Python, it is (at least currently) the same
 version of Python being built.  Therefore I wrote about the bootstrapping
 problems when bytecode changes.

 Depending on Cython is better in that it breaks the bootstrapping cycle,
 but on the other hand the C code may need to be regenerated when the C
>>> API
 changes in an incompatible way.
>>>
>>> Cython OTOH probably needs Python 2.x, which isn't that great for
>>> building Python 3. And requiring Cython for developing is not very
>>> contributor-friendly.
>>>
>>
>> Well, required to regenerate _frozen_importlib, but nothing else. I mean
>> making fixes go into importlib directly and get tested that way, not
>> through __import__(). So really Cython would only be needed when
>> importlib._bootstrap has been changed and you are making a commit.
> 
> That's still a large dependency to bring in, while we already have a
> working solution.
> I'd understand using Cython to develop some new extension module which
> requires linking against a C library (and is thus impossible to write
> in pure Python). But for importlib that's totally non-necessary.
> 
> I guess I'm -1 on it.

I agree. If the problem we're trying to solve is that the generated file
isn't always rebuilt, bringing in a large dependency like Cython seems
like overkill to me.

We basically have a working solution now (thanks, Brett). I think we
should focus on getting it polished. Maybe we can bring in Cython in a
later release, if in the 3.4 timeframe it still seems like we have a
problem to solve. I suspect things will be working fine.

Eric.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Antoine Pitrou]

On Mon, 16 Apr 2012 20:41:56 -0400
Brett Cannon  wrote:
> On Mon, Apr 16, 2012 at 20:27, Antoine Pitrou  wrote:
> 
> > On Tue, 17 Apr 2012 01:11:14 +0200
> > Georg Brandl  wrote:
> > >
> > > No, it's not just an existing Python, it is (at least currently) the same
> > > version of Python being built.  Therefore I wrote about the bootstrapping
> > > problems when bytecode changes.
> > >
> > > Depending on Cython is better in that it breaks the bootstrapping cycle,
> > > but on the other hand the C code may need to be regenerated when the C
> > API
> > > changes in an incompatible way.
> >
> > Cython OTOH probably needs Python 2.x, which isn't that great for
> > building Python 3. And requiring Cython for developing is not very
> > contributor-friendly.
> >
> 
> Well, required to regenerate _frozen_importlib, but nothing else. I mean
> making fixes go into importlib directly and get tested that way, not
> through __import__(). So really Cython would only be needed when
> importlib._bootstrap has been changed and you are making a commit.

That's still a large dependency to bring in, while we already have a
working solution.
I'd understand using Cython to develop some new extension module which
requires linking against a C library (and is thus impossible to write
in pure Python). But for importlib that's totally non-necessary.

I guess I'm -1 on it.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Stefan Behnel]

Antoine Pitrou, 17.04.2012 02:27:
> On Tue, 17 Apr 2012 01:11:14 +0200
> Georg Brandl wrote:
>> No, it's not just an existing Python, it is (at least currently) the same
>> version of Python being built.  Therefore I wrote about the bootstrapping
>> problems when bytecode changes.
>>
>> Depending on Cython is better in that it breaks the bootstrapping cycle,
>> but on the other hand the C code may need to be regenerated when the C API
>> changes in an incompatible way.
> 
> Cython OTOH probably needs Python 2.x, which isn't that great for
> building Python 3.

It uses 2to3 at install time, so you get a Py3 version out of it. No need
to have Py2 installed in order to use it.


> And requiring Cython for developing is not very
> contributor-friendly.

Brett Cannon answered that one. If you ship the C sources, developers will
only be impacted when they want to modify source code that gets compiled
with Cython.

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Stefan Behnel]

Amaury Forgeot d'Arc, 16.04.2012 22:43:
> 2012/4/16 Stefan Behnel
>>> On the down side, we would depend on Cython for evolving .
>>
>> Right, although not as a strict dependency. The code would still work just
>> fine in plain Python.
> 
> Not quite, we are talking of the imp module here...

Hmm, right, after writing the above, I figured that it would at least have
to do something like "import sys" in order to deal with the import config
(path, meta path, ...). That obviously won't work in pure Python at this point.

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Brett Cannon]

On Mon, Apr 16, 2012 at 20:27, Antoine Pitrou  wrote:

> On Tue, 17 Apr 2012 01:11:14 +0200
> Georg Brandl  wrote:
> >
> > No, it's not just an existing Python, it is (at least currently) the same
> > version of Python being built.  Therefore I wrote about the bootstrapping
> > problems when bytecode changes.
> >
> > Depending on Cython is better in that it breaks the bootstrapping cycle,
> > but on the other hand the C code may need to be regenerated when the C
> API
> > changes in an incompatible way.
>
> Cython OTOH probably needs Python 2.x, which isn't that great for
> building Python 3. And requiring Cython for developing is not very
> contributor-friendly.
>

Well, required to regenerate _frozen_importlib, but nothing else. I mean
making fixes go into importlib directly and get tested that way, not
through __import__(). So really Cython would only be needed when
importlib._bootstrap has been changed and you are making a commit.

-Brett


>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Antoine Pitrou]

On Tue, 17 Apr 2012 01:11:14 +0200
Georg Brandl  wrote:
> 
> No, it's not just an existing Python, it is (at least currently) the same
> version of Python being built.  Therefore I wrote about the bootstrapping
> problems when bytecode changes.
> 
> Depending on Cython is better in that it breaks the bootstrapping cycle,
> but on the other hand the C code may need to be regenerated when the C API
> changes in an incompatible way.

Cython OTOH probably needs Python 2.x, which isn't that great for
building Python 3. And requiring Cython for developing is not very
contributor-friendly.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Georg Brandl]

On 16.04.2012 18:15, R. David Murray wrote:

On Mon, 16 Apr 2012 11:21:34 -0400, Brett Cannon  wrote:

 On Mon, Apr 16, 2012 at 10:07, "Martin v. Löwis"  wrote:

 >  >  We have other instances of this (e.g. the Objects/typeslots.inc file
 >  >  is generated and checked in), but in the case of importlib, we have
 >  >  to use the ./python binary for freezing to avoid bytecode
 >  >  incompatibilities, which obviously is a problem if ./python isn't
 >  >  built yet.
 >
 >  As for dependencies on byte code: we could consider using Cython instead
 >  of freeze (not sure whether Cython would build the bootstrap correctly;
 >  it may need to be fixed first). With that, we would get semi-readable
 >  source code, which should also play more nicely with hg diffs. On the
 >  down side, we would depend on Cython for evolving .


That is an interesting idea.  We already depend on external tools, e.g.
autotools, for updating checked-in files, so why not Cython.


 We could also just store the raw source code and use that if we are all
 willing to pay the performance cost of parsing and compiling the code at
 every startup.


I don't see how depending on Cython is better than depending on having
an existing Python.


No, it's not just an existing Python, it is (at least currently) the same
version of Python being built.  Therefore I wrote about the bootstrapping
problems when bytecode changes.

Depending on Cython is better in that it breaks the bootstrapping cycle,
but on the other hand the C code may need to be regenerated when the C API
changes in an incompatible way.


 If the only benefit is semi-readable code, surely
we do have source code for the pre-frozen module, and it is just a matter
of convincing hg that the bytecode is binary, not text?


The benefit is also (presumably) better performance.

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Martin v. Löwis]

Am 16.04.2012 19:44, schrieb Antoine Pitrou:
> On Mon, 16 Apr 2012 13:33:45 -0400
> Brett Cannon  wrote:
>> On Mon, Apr 16, 2012 at 13:08, "Martin v. Löwis"  wrote:
>>
 So like execute hg diff on the dependent files and if nothing changed
 then touch the auto-generated file w/ 'touch' to prevent future attempts
 to execute the target?
>>>
>>> Exactly. There might be something better than hg diff, perhaps some form
>>> of hg status.
>>>
>>
>> Yeah, hg status is probably better. Now someone just needs to write the
>> shell script. =)
> 
> Wouldn't it be better if Python could compile regardless of the
> presence of a hg repository?

Who says it couldn't?

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Amaury Forgeot d'Arc]

Hi,

2012/4/16 Stefan Behnel 

> > On the down side, we would depend on Cython for evolving .
>
> Right, although not as a strict dependency. The code would still work just
> fine in plain Python.


Not quite, we are talking of the imp module here...

-- 
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Stefan Behnel]

"Martin v. Löwis", 16.04.2012 16:07:
>> We have other instances of this (e.g. the Objects/typeslots.inc file
>> is generated and checked in), but in the case of importlib, we have
>> to use the ./python binary for freezing to avoid bytecode
>> incompatibilities, which obviously is a problem if ./python isn't
>> built yet.
> 
> As for dependencies on byte code: we could consider using Cython instead
> of freeze (not sure whether Cython would build the bootstrap correctly;
> it may need to be fixed first).

I assume that this would be needed rather early during the interpreter
startup, so you're right that there may be obstacles due to expectations in
the generated C code that may not be there yet. But I think it's worth a
try. Cython is rather predictable in its requirements given the Python
source code. Just run the module through Cython and see what it gives you.
(Note that you may need the latest github master version to make everything
work nicely in Py3.3, I keep adapting things there as I find them - our CI
server runs daily tests against the latest CPython master.)

One thing Cython does during module initialisation is to import the
builtins module using PyModule_AddModule(). Does that work already at this
point? It uses it mostly to cache builtins that the module uses. That can
be disabled, though, and I also wouldn't mind letting Cython put a
preprocessor guard into that code that would let it do something else based
on a macro that CPython defines at this point, maybe even just
Py_BUILD_CORE. We already do these things for all sorts of purposes.

And, obviously, the module init function can also be called directly
instead of running it as part of an import. That's commonly done when using
Cython modules in an embedded Python runtime.


> With that, we would get semi-readable
> source code, which should also play more nicely with hg diffs.

There's still a Cython patch hanging around on github that aims to keep the
C code from changing all that drastically on Python source code changes
(e.g. due to source line references etc.). Might be worth integrating for
something like this. There's also a switch that disables the (helpful)
reproduction of the Python code context in C code comments, in case that
gets in the way for diffs.


> On the down side, we would depend on Cython for evolving .

Right, although not as a strict dependency. The code would still work just
fine in plain Python. But it would depend on Cython for performance. And we
usually recommend to ship the generated C sources anyway to avoid a user
dependency on Cython, so this use case is quite normal.

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[R. David Murray]

On Mon, 16 Apr 2012 13:51:35 -0400, Barry Warsaw  wrote:
> On Apr 16, 2012, at 07:44 PM, Antoine Pitrou wrote:
> 
> >Wouldn't it be better if Python could compile regardless of the
> >presence of a hg repository?
> 
> If you want it in your $DISTRO, yes please!

My impression is that our usual solution for this is to make sure the
timestamps are correct in distributed tarballs, so that the hg-dependent
machinery is not invoked when building from a release tarball.  Is this
case any different?

--David
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Barry Warsaw]

On Apr 16, 2012, at 07:44 PM, Antoine Pitrou wrote:

>Wouldn't it be better if Python could compile regardless of the
>presence of a hg repository?

If you want it in your $DISTRO, yes please!

-Barry
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Antoine Pitrou]

On Mon, 16 Apr 2012 13:33:45 -0400
Brett Cannon  wrote:
> On Mon, Apr 16, 2012 at 13:08, "Martin v. Löwis"  wrote:
> 
> > > So like execute hg diff on the dependent files and if nothing changed
> > > then touch the auto-generated file w/ 'touch' to prevent future attempts
> > > to execute the target?
> >
> > Exactly. There might be something better than hg diff, perhaps some form
> > of hg status.
> >
> 
> Yeah, hg status is probably better. Now someone just needs to write the
> shell script. =)

Wouldn't it be better if Python could compile regardless of the
presence of a hg repository?

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Eric Snow]

On Mon, Apr 16, 2012 at 11:32 AM, Brett Cannon  wrote:
>
>
> On Mon, Apr 16, 2012 at 13:04, "Martin v. Löwis"  wrote:
>>
>> > I don't see how depending on Cython is better than depending on having
>> > an existing Python.  If the only benefit is semi-readable code, surely
>> > we do have source code for the pre-frozen module, and it is just a
>> > matter
>> > of convincing hg that the bytecode is binary, not text?
>>
>> Cython-generated C code would likely be more stable (and produce
>> compiler errors if it gets stale), whereas importlib.h needs to be
>> regenerated with byte code changes.
>>
>> Having source code has the advantage that it becomes possible to
>> single-step through the import process in C debugger. Single-stepping
>> with pdb would, of course, be better than that, but I doubt it's
>> feasible.
>>
>> In addition, there might be a performance gain with Cython over ceval.
>
>
> The other benefit is maintainability. In order to hit my roughly 5% startup
> speed I had to rewrite chunks of __import__() in C code and then delegate to
> importlib's Python code in cases where sys.modules was not hit. Using Cython
> would mean that can all go away and the differences between the C and Python
> code would become (supposedly) non-existent, making tweaks easier (e.g. when
> I made the change to hit sys.modules less when a loader returned the desired
> module it was annoying to have to change importlib *and* import.c).

+1 on reducing the complexity of the import code.

-eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Brett Cannon]

On Mon, Apr 16, 2012 at 13:08, "Martin v. Löwis"  wrote:

> > So like execute hg diff on the dependent files and if nothing changed
> > then touch the auto-generated file w/ 'touch' to prevent future attempts
> > to execute the target?
>
> Exactly. There might be something better than hg diff, perhaps some form
> of hg status.
>

Yeah, hg status is probably better. Now someone just needs to write the
shell script. =)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Brett Cannon]

On Mon, Apr 16, 2012 at 13:04, "Martin v. Löwis"  wrote:

> > I don't see how depending on Cython is better than depending on having
> > an existing Python.  If the only benefit is semi-readable code, surely
> > we do have source code for the pre-frozen module, and it is just a matter
> > of convincing hg that the bytecode is binary, not text?
>
> Cython-generated C code would likely be more stable (and produce
> compiler errors if it gets stale), whereas importlib.h needs to be
> regenerated with byte code changes.
>
> Having source code has the advantage that it becomes possible to
> single-step through the import process in C debugger. Single-stepping
> with pdb would, of course, be better than that, but I doubt it's
> feasible.
>
> In addition, there might be a performance gain with Cython over ceval.
>

The other benefit is maintainability. In order to hit my roughly 5% startup
speed I had to rewrite chunks of __import__() in C code and then delegate
to importlib's Python code in cases where sys.modules was not hit. Using
Cython would mean that can all go away and the differences between the C
and Python code would become (supposedly) non-existent, making tweaks
easier (e.g. when I made the change to hit sys.modules less when a loader
returned the desired module it was annoying to have to change importlib
*and* import.c).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Martin v. Löwis]

> So like execute hg diff on the dependent files and if nothing changed
> then touch the auto-generated file w/ 'touch' to prevent future attempts
> to execute the target?

Exactly. There might be something better than hg diff, perhaps some form
of hg status.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Martin v. Löwis]

> I don't see how depending on Cython is better than depending on having
> an existing Python.  If the only benefit is semi-readable code, surely
> we do have source code for the pre-frozen module, and it is just a matter
> of convincing hg that the bytecode is binary, not text?

Cython-generated C code would likely be more stable (and produce
compiler errors if it gets stale), whereas importlib.h needs to be
regenerated with byte code changes.

Having source code has the advantage that it becomes possible to
single-step through the import process in C debugger. Single-stepping
with pdb would, of course, be better than that, but I doubt it's
feasible.

In addition, there might be a performance gain with Cython over ceval.

Regards,
Martin

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Brett Cannon]

On Mon, Apr 16, 2012 at 12:31, Antoine Pitrou  wrote:

> On Mon, 16 Apr 2012 12:15:16 -0400
> "R. David Murray"  wrote:
> >
> > I don't see how depending on Cython is better than depending on having
> > an existing Python.  If the only benefit is semi-readable code, surely
> > we do have source code for the pre-frozen module, and it is just a matter
> > of convincing hg that the bytecode is binary, not text?
> >
> > Brett's earlier thought of compiling from source as a *fallback* makes
> > sense to me.  I'd rather not add overhead to startup that we can avoid.
>
>
In reply to David, one trick with this, though, is that frozen modules
don't store the magic number of the bytecode, so that would need to change
in order to make this fully feasible.


> Compiling from source at which point, though?
>

At startup of the interpreter.


> In essence, that would mean reimplement Python/freeze_importlib.py in C?
> We could even compile it to a separate executable that gets built
> before the Python executable (like pgen) :-)
>

So a mini Python that just knew how to compile to bytecode and nothing more?

-Brett


>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Antoine Pitrou]

On Mon, 16 Apr 2012 12:15:16 -0400
"R. David Murray"  wrote:
> 
> I don't see how depending on Cython is better than depending on having
> an existing Python.  If the only benefit is semi-readable code, surely
> we do have source code for the pre-frozen module, and it is just a matter
> of convincing hg that the bytecode is binary, not text?
> 
> Brett's earlier thought of compiling from source as a *fallback* makes
> sense to me.  I'd rather not add overhead to startup that we can avoid.

Compiling from source at which point, though?
In essence, that would mean reimplement Python/freeze_importlib.py in C?
We could even compile it to a separate executable that gets built
before the Python executable (like pgen) :-)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[R. David Murray]

On Mon, 16 Apr 2012 11:21:34 -0400, Brett Cannon  wrote:
> On Mon, Apr 16, 2012 at 10:07, "Martin v. Löwis"  wrote:
> 
> > > We have other instances of this (e.g. the Objects/typeslots.inc file
> > > is generated and checked in), but in the case of importlib, we have
> > > to use the ./python binary for freezing to avoid bytecode
> > > incompatibilities, which obviously is a problem if ./python isn't
> > > built yet.
> >
> > As for dependencies on byte code: we could consider using Cython instead
> > of freeze (not sure whether Cython would build the bootstrap correctly;
> > it may need to be fixed first). With that, we would get semi-readable
> > source code, which should also play more nicely with hg diffs. On the
> > down side, we would depend on Cython for evolving .
> >
> 
> We could also just store the raw source code and use that if we are all
> willing to pay the performance cost of parsing and compiling the code at
> every startup.

I don't see how depending on Cython is better than depending on having
an existing Python.  If the only benefit is semi-readable code, surely
we do have source code for the pre-frozen module, and it is just a matter
of convincing hg that the bytecode is binary, not text?

Brett's earlier thought of compiling from source as a *fallback* makes
sense to me.  I'd rather not add overhead to startup that we can avoid.

--David
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Brett Cannon]

On Mon, Apr 16, 2012 at 10:07, "Martin v. Löwis"  wrote:

> > We have other instances of this (e.g. the Objects/typeslots.inc file
> > is generated and checked in), but in the case of importlib, we have
> > to use the ./python binary for freezing to avoid bytecode
> > incompatibilities, which obviously is a problem if ./python isn't
> > built yet.
>
> As for dependencies on byte code: we could consider using Cython instead
> of freeze (not sure whether Cython would build the bootstrap correctly;
> it may need to be fixed first). With that, we would get semi-readable
> source code, which should also play more nicely with hg diffs. On the
> down side, we would depend on Cython for evolving .
>

We could also just store the raw source code and use that if we are all
willing to pay the performance cost of parsing and compiling the code at
every startup.


>
> As for the timestamp issue: I think we should add a target "make touch"
> or some such which checks whether the respective files are unmodified
> in the local clone, and if so, arranges to touch generated files that
> are older than their sources. If this is done by a plain shell script,
> one could also make this a post-update Mercurial hook.
>

So like execute hg diff on the dependent files and if nothing changed then
touch the auto-generated file w/ 'touch' to prevent future attempts to
execute the target?

-Brett


>
> Regards,
> Martin
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Martin v. Löwis]

> We have other instances of this (e.g. the Objects/typeslots.inc file
> is generated and checked in), but in the case of importlib, we have
> to use the ./python binary for freezing to avoid bytecode
> incompatibilities, which obviously is a problem if ./python isn't
> built yet.

As for dependencies on byte code: we could consider using Cython instead
of freeze (not sure whether Cython would build the bootstrap correctly;
it may need to be fixed first). With that, we would get semi-readable
source code, which should also play more nicely with hg diffs. On the
down side, we would depend on Cython for evolving .

As for the timestamp issue: I think we should add a target "make touch"
or some such which checks whether the respective files are unmodified
in the local clone, and if so, arranges to touch generated files that
are older than their sources. If this is done by a plain shell script,
one could also make this a post-update Mercurial hook.

Regards,
Martin

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Stefan Behnel]

Antoine Pitrou, 16.04.2012 13:13:
> On Mon, 16 Apr 2012 09:54:41 +0200
> Stefan Behnel wrote:
>>
>> The new import cache broke Cython's load of on-the-fly compiled extension
>> modules, which naively used "__import__(module_name)" after building them.
>> I could fix that by moving to "imp.load_dynamic()" (we know where we put
>> the compiled module anyway), although I just noticed that that's not
>> actually documented. So I hope that won't break later on.
> 
> You can call importlib.invalidate_caches().
> http://docs.python.org/dev/library/importlib.html#importlib.invalidate_caches

Well, yes, but imp.load_dynamic() would be the right thing to do for us. Is
there a reason why it's not documented?

I would like to avoid changing the code to load_dynamic() now and then
having to realise that that's going to die in 3.3 final because it somehow
got in the way of the importlob rewrites and is not being considered a
valuable enough public API.

New doc bug ticket:

http://bugs.python.org/issue14594


>> The next thing I noticed is that the old-style level -1 import no longer
>> works, which presumably breaks a *lot* of Cython compiled modules. It used
>> to work in the master branch until two days ago, now it raises a
>> ValueError. We may be able to fix this by copying over CPython's old import
>> code into Cython, but I actually wonder if this was really intended. If
>> this feature wasn't deliberately broken in Py3.0, why break it now?
> 
> Regressions should be reported on the bug tracker IMHO.

It was meant as more of a question for now, but here it goes:

http://bugs.python.org/issue14592

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Antoine Pitrou]

On Mon, 16 Apr 2012 09:54:41 +0200
Stefan Behnel  wrote:
> 
> The new import cache broke Cython's load of on-the-fly compiled extension
> modules, which naively used "__import__(module_name)" after building them.
> I could fix that by moving to "imp.load_dynamic()" (we know where we put
> the compiled module anyway), although I just noticed that that's not
> actually documented. So I hope that won't break later on.

You can call importlib.invalidate_caches().
http://docs.python.org/dev/library/importlib.html#importlib.invalidate_caches

> The next thing I noticed is that the old-style level -1 import no longer
> works, which presumably breaks a *lot* of Cython compiled modules. It used
> to work in the master branch until two days ago, now it raises a
> ValueError. We may be able to fix this by copying over CPython's old import
> code into Cython, but I actually wonder if this was really intended. If
> this feature wasn't deliberately broken in Py3.0, why break it now?

Regressions should be reported on the bug tracker IMHO.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Stefan Behnel]

Brett Cannon, 14.04.2012 20:12:
> My multi-year project -- started in 2006 according to my blog -- to rewrite
> import in pure Python and then bootstrap it into CPython as *the*
> implementation of __import__() is finally over (mostly)! Hopefully I didn't
> break too much code in the process. =)

Well, some at least.

The new import cache broke Cython's load of on-the-fly compiled extension
modules, which naively used "__import__(module_name)" after building them.
I could fix that by moving to "imp.load_dynamic()" (we know where we put
the compiled module anyway), although I just noticed that that's not
actually documented. So I hope that won't break later on.

The next thing I noticed is that the old-style level -1 import no longer
works, which presumably breaks a *lot* of Cython compiled modules. It used
to work in the master branch until two days ago, now it raises a
ValueError. We may be able to fix this by copying over CPython's old import
code into Cython, but I actually wonder if this was really intended. If
this feature wasn't deliberately broken in Py3.0, why break it now?

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib is now bootstrapped (and what that means)

[Georg Brandl]

On 14.04.2012 20:12, Brett Cannon wrote:

My multi-year project -- started in 2006 according to my blog -- to rewrite
import in pure Python and then bootstrap it into CPython as *the* implementation
of __import__() is finally over (mostly)! Hopefully I didn't break too much code
in the process. =)

Now this is "mostly" finished because the single incompatibility that importlib
has is that it doesn't check the Windows registry for paths to search since I
lack a Windows installation to develop and test on. If someone can tackle that
issue that would be greatly appreciated (http://bugs.python.org/issue14578).

Next up is how to maintain/develop for all of this. So the Makefile will
regenerate Python/importlib.h whenever Lib/importlib/_bootstrap.py or
Python/freeze_importlib.py changes.  So if you make a change to importlib make
sure to get it working and tested before running 'make' again else you will
generate a bad frozen importlib (if you do mess up you can also revert the
changes to importlib.h and re-run make; a perk to having importlib.h checked
in). Otherwise keep in mind that you can't use any module that isn't a builtin
(sys.builtin_module_names) in importlib._bootstrap since you can't import
something w/o import working. =)


We've just now talked on IRC about this regeneration.  Since both files --
_bootstrap.py and importlib.h -- are checked in, a make run can try to re-
generate importlib.h.  This depends on the timestamps of the two files, which I 
don't think Mercurial makes any guarantees about.


We have other instances of this (e.g. the Objects/typeslots.inc file is 
generated and checked in), but in the case of importlib, we have to use the

./python binary for freezing to avoid bytecode incompatibilities, which
obviously is a problem if ./python isn't built yet.


And the only outstanding point of contention in all of this is that some people
don't like having freeze_importlib.py in Python/ and instead want it in Tools/.
I didn't leave it in Tools/ as I have always viewed that Python should build w/o
the Tools directory, but maybe the Linux distros actually ship with it and thus
this is an unneeded worry. Plus the scripts to generate the AST are in Parser so
there is precedent for what I have done.


I would have no objections to Python/.  There is also e.g. Objects/typeslots.py.

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] importlib is now bootstrapped (and what that means)

[Brett Cannon]

My multi-year project -- started in 2006 according to my blog -- to rewrite
import in pure Python and then bootstrap it into CPython as *the*
implementation of __import__() is finally over (mostly)! Hopefully I didn't
break too much code in the process. =)

Now this is "mostly" finished because the single incompatibility that
importlib has is that it doesn't check the Windows registry for paths to
search since I lack a Windows installation to develop and test on. If
someone can tackle that issue that would be greatly appreciated (
http://bugs.python.org/issue14578).

Next up is how to maintain/develop for all of this. So the Makefile will
regenerate Python/importlib.h whenever Lib/importlib/_bootstrap.py or
Python/freeze_importlib.py changes. So if you make a change to importlib
make sure to get it working and tested before running 'make' again else you
will generate a bad frozen importlib (if you do mess up you can also revert
the changes to importlib.h and re-run make; a perk to having importlib.h
checked in). Otherwise keep in mind that you can't use any module that
isn't a builtin (sys.builtin_module_names) in importlib._bootstrap since
you can't import something w/o import working. =)

Where does this leave imp and Python/import.c? I want to make imp into _imp
and then implement as much as possible in pure Python (either in importlib
itself or in Lib/imp.py). Once that has happened then more C code in
import.c can be gutted (see http://bugs.python.org/issue13959 for tracking
this work which I will start piecemeal shortly).

I have some ideas on how to improve things for import, but I'm going to do
them as separate emails to have separate discussion threads on them so all
of this is easier to follow (e.g. actually following through on PEP 302 and
exposing the import machinery as importers instead of having anything be
implicit, etc.).

And the only outstanding point of contention in all of this is that some
people don't like having freeze_importlib.py in Python/ and instead want it
in Tools/. I didn't leave it in Tools/ as I have always viewed that Python
should build w/o the Tools directory, but maybe the Linux distros actually
ship with it and thus this is an unneeded worry. Plus the scripts to
generate the AST are in Parser so there is precedent for what I have done.

Anyway, I will write up the What's New entry and double-check the language
spec for updating once all of the potential changes I want to talk about in
other emails have been resolved.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib cleared for merging into default!

[Paul Moore]

On 7 March 2012 23:05, Brett Cannon  wrote:
> At the language summit today I got clearance to merge my importlib bootstrap
> branch (http://hg.python.org/sandbox/bcannon#bootstrap_importlib) thanks to
> performance being about 5% slower using the normal_startup (which, as Thomas
> Wouters said, is less than the difference of using the newest gcc in some
> benchmarking he recently did). Obviously thanks to everyone who has helped
> out with making this happen over the years!

Yay! Congratulations for getting this done. When I first co-authored
PEP 302 I never realised what a long road it would be from there to
here. Thanks for making it happen.

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] importlib cleared for merging into default!

[Brett Cannon]

At the language summit today I got clearance to merge my importlib
bootstrap branch (http://hg.python.org/sandbox/bcannon#bootstrap_importlib)
thanks to performance being about 5% slower using the normal_startup
(which, as Thomas Wouters said, is less than the difference of using the
newest gcc in some benchmarking he recently did). Obviously thanks to
everyone who has helped out with making this happen over the years!

So, where does that leave us? First is getting full code review and sign
off from somebody (http://bugs.python.org/issue2377 is the issue tracking
this). Once I have that I will do the merge and then go through the tracker
to update issues to see if they still apply or need to be re-targeted for
importlib.

Once importlib has been merged there is some stuff that needs to happen. I
will first strip imp down to what has to be in import.c (e.g. importing a
built-in module), rename it _imp, and then re-implement/extend whatever is
needed in an imp.py module. This will allow for much of the C code left in
import.c to go away (i.e. imp.find_module() is the reason the finder code
in import.c has not been removed yet). It will also alleviate the other VMs
from having to implement all of imp from scratch.

Once importlib is in I will also do some proposing on how to undo some
import design decisions that were caused because it was all C code (e.g.
implicit stuff like a sys.meta_path entry that handles
sys.path/sys.path_importer_cache/sys.path_hooks, the meaning of None in
sys.path_importer_cache). Everyone at the language summit was supportive of
this stuff and basically agreed to it but wanted it as a separate step from
the merge to get everything moving faster.

What can be done in parallel/while waiting is exposing more of importlib's
innards. This ties into some of the proposals I will be making once the
merge occurs.

Everything else I have in mind is long term stdlib cleanup using importlib,
but that is not important now.

IOW, someone please review my bootstrap_importlib branch so I can merge it.
=)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib quest

[Brett Cannon]

On Mon, Feb 6, 2012 at 14:49, Antoine Pitrou  wrote:

> On Mon, 6 Feb 2012 09:57:56 -0500
> Brett Cannon  wrote:
> > Thanks for any help people can provide me on this now 5 year quest to get
> > this work finished.
>
> Do you have any plan to solve the performance issue?
>

I have not even looked at performance or attempted to profile the code, so
I suspect there is room for improvement.


>
> $ ./python -m timeit -s "import sys; mod='struct'" \
>  "__import__(mod); del sys.modules[mod]"
> 1 loops, best of 3: 75.3 usec per loop
> $ ./python -m timeit -s "import sys; mod='struct'; from importlib import
> __import__" \
>  "__import__(mod); del sys.modules[mod]"
> 1000 loops, best of 3: 421 usec per loop
>
> Startup time is already much worse in 3.3 than in 2.7. With such a
> slowdown in importing fresh modules, applications using many batteries
> (third-party or not) will be heavily impacted.
>

I have a benchmark suite for importing modules directly at
importlib.test.benchmark, but it doesn't explicitly cover searching far
down sys.path. I will see if any of the existing tests implicitly do that
and if not add it.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib quest

[Antoine Pitrou]

On Mon, 6 Feb 2012 20:49:48 +0100
Antoine Pitrou  wrote:

> On Mon, 6 Feb 2012 09:57:56 -0500
> Brett Cannon  wrote:
> > Thanks for any help people can provide me on this now 5 year quest to get
> > this work finished.
> 
> Do you have any plan to solve the performance issue?
> 
> $ ./python -m timeit -s "import sys; mod='struct'" \
>   "__import__(mod); del sys.modules[mod]"
> 1 loops, best of 3: 75.3 usec per loop
> $ ./python -m timeit -s "import sys; mod='struct'; from importlib import 
> __import__" \
>   "__import__(mod); del sys.modules[mod]"
> 1000 loops, best of 3: 421 usec per loop

The culprit for the overhead is likely to be PathFinder.find_module:

$ ./python -m timeit -s "import sys; mod='struct'; from importlib._bootstrap 
import _DefaultPathFinder; finder=_DefaultPathFinder" 
"finder.find_module('struct')"
1000 loops, best of 3: 355 usec per loop
$ ./python -S -m timeit -s "import sys; mod='struct'; from importlib._bootstrap 
import _DefaultPathFinder; finder=_DefaultPathFinder" 
"finder.find_module('struct')"
1 loops, best of 3: 176 usec per loop

Note how it's dependent on sys.path length. On an installed Python with
many additional sys.path entries (e.g. because of distribute-based
module installs), import times will be much worse.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] importlib quest

[Antoine Pitrou]

On Mon, 6 Feb 2012 09:57:56 -0500
Brett Cannon  wrote:
> Thanks for any help people can provide me on this now 5 year quest to get
> this work finished.

Do you have any plan to solve the performance issue?

$ ./python -m timeit -s "import sys; mod='struct'" \
  "__import__(mod); del sys.modules[mod]"
1 loops, best of 3: 75.3 usec per loop
$ ./python -m timeit -s "import sys; mod='struct'; from importlib import 
__import__" \
  "__import__(mod); del sys.modules[mod]"
1000 loops, best of 3: 421 usec per loop

Startup time is already much worse in 3.3 than in 2.7. With such a
slowdown in importing fresh modules, applications using many batteries
(third-party or not) will be heavily impacted.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib

[Brett Cannon]

I have updated the benchmark to now measure importing source w/o writing
bytecode, importing source & writing bytecode, and importing bytecode w/
source (as I don't care about sourceless import performance).

Now, before you look at these numbers, realize that I have not once tried to
profile importlib to see where its hot spots are (only optimization I have
done is cut down on the stat calls since Python 3.1 when I re-developed the
ABCs). I'm sure if I profiled the code and wrote key bits in C these
performance numbers would improve reasonably quickly.

Anyway, on my 2.2 GHz MacBook, this leads to::

import.c
sys.modules [ 223337 223036 223362 ] best is 223362
Built-in module [ 23347 23319 23331 ] best is 23347
Bytecode w/ source [ 6624 6607 6608 ] best is 6624
Source w/o bytecode [ 4643 4674 4655 ] best is 4674
Source writing bytecode [ 2063 2145 2204 ] best is 2204

importlib
sys.modules [ 43423 43414 43426 ] best is 43426
Built-in module [ 9130 9115 9120 ] best is 9130
Bytecode w/ source [ 1554 1556 1556 ] best is 1556
Source w/o bytecode [ 1351 1351 1353 ] best is 1353
Source writing bytecode [ 786 843 810 ] best is 843

importlib / import.c:
sys.modules  19%
Built-in module  39%
Bytecode w/ source  23%
Source w/o bytecode  29%
Source writing bytecode  38%

What does this show? Stuff that requires a lot of I/O has the smallest
performance difference (source writing bytecode), but where there is as
little I/O as possible (bytecode w/ source) import.c wins as it has to do
less. This is also why sys.modules is so damn fast; it's the smallest amount
of C code you can run while importlib has standard Python calling overhead.

It should also be pointed out that importlib has fully implemented PEP 302
and intentionally has the loaders using their own exposed PEP 302 APIs. This
means there are a lot more methods calls than in the C version, along with a
lot less corners cut in the name of performance. So while importlib will be
slower simply because it's implemented in C, it will also be slower because
the darn thing is actually written to follow the PEPs we have along with
making it easier for people to subclass and benefit from the import code.

Anyway, as I have said, I need to hit 100% compatibility when running the
test suite -- run importlib.test.regrtest to see where it fails now; also
read that file as it has notes as to why the known failures are happening --
before I start worrying about bootstrapping and performance and that will
all be no sooner than Python 3.3.

On Thu, Jul 15, 2010 at 04:55, Nick Coghlan  wrote:

> On Thu, Jul 15, 2010 at 4:06 PM, Brett Cannon  wrote:
> >> In any case, here my results under a Linux system:
> >>
> >> $ ./python -m importlib.test.benchmark
> >> sys.modules [ 323782 326183 326667 ] best is 326667
> >> Built-in module [ 33600 33693 33610 ] best is 33693
> >>
> >> $ ./python -m importlib.test.benchmark -b
> >> sys.modules [ 1297640 1315366 1292283 ] best is 1315366
> >> Built-in module [ 58180 57708 58057 ] best is 58180
> >
> > And this is what might make evaluating importlib tough; while the
> > performance is 25% of what it is for import.c, being able to import
> > over 300,000 times/second is still damn fast.
>
> Yeah, I think the numbers where the filesystem gets involved are going
> to be more relevant. Modules that have already been cached and those
> built in to the executable aren't likely to dominate interpreter and
> application startup times (which is the main thing I'm worried about
> seeing regress).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib

[Antoine Pitrou]

On Wed, 14 Jul 2010 23:06:58 -0700
Brett Cannon  wrote:
> >
> > In any case, here my results under a Linux system:
> >
> > $ ./python -m importlib.test.benchmark
> > sys.modules [ 323782 326183 326667 ] best is 326667
> > Built-in module [ 33600 33693 33610 ] best is 33693
> >
> > $ ./python -m importlib.test.benchmark -b
> > sys.modules [ 1297640 1315366 1292283 ] best is 1315366
> > Built-in module [ 58180 57708 58057 ] best is 58180
> 
> And this is what might make evaluating importlib tough; while the
> performance is 25% of what it is for import.c, being able to import
> over 300,000 times/second is still damn fast.

Yes, that's very encouraging.
I guess the final test would be to take something like Mercurial, and
time e.g. "hg version" both with the builtin-import, and with importlib
enabled as default import mechanism.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib

[John Nagle]

anatoly techtonik wrote:

What about web-applications? Is that true that for FastCgi or mod_wsgi
deamon mode interpreter and application is started only once per say
100 requests?


   Yes.  Only CGI programs reload on every use.  FCGI/WSGI programs
run more or less forever, and are only loaded once, reading
requests and handling them.  Some FCGI systems reload their applications 
once in a while, and ask the idle ones to exit

after periods of no traffic, but that's a form of automatic
performance tuning. And, of course, there are the permanently
resident systems like Twisted.

   (Incidentally, if you run FCGI/WSGI programs under Apache,
make sure that you don't put .fcgi scripts in a directory
from which Apache will execute .cgi scripts. Apache will run
the .fcgi script with CGI, and the WSGI Python module will
detect this and run the script one time and exit for each
request.  Everything appears to work and performance is terrible.)

   If import performance is important in a production server
environment, it's more useful to look at minimizing the number
of loads rather than trying to make them faster.  If and when
Unladen Swallow finally gets going, startup times will get worse
as the JIT compiler cranks, but run time will (hopefully) decrease.

John Nagle

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib

[anatoly techtonik]

On Thu, Jul 15, 2010 at 2:55 PM, Nick Coghlan  wrote:
>>> In any case, here my results under a Linux system:
>>>
>>> $ ./python -m importlib.test.benchmark
>>> sys.modules [ 323782 326183 326667 ] best is 326667
>>> Built-in module [ 33600 33693 33610 ] best is 33693
>>>
>>> $ ./python -m importlib.test.benchmark -b
>>> sys.modules [ 1297640 1315366 1292283 ] best is 1315366
>>> Built-in module [ 58180 57708 58057 ] best is 58180
>>
>> And this is what might make evaluating importlib tough; while the
>> performance is 25% of what it is for import.c, being able to import
>> over 300,000 times/second is still damn fast.
>
> Yeah, I think the numbers where the filesystem gets involved are going
> to be more relevant. Modules that have already been cached and those
> built in to the executable aren't likely to dominate interpreter and
> application startup times (which is the main thing I'm worried about
> seeing regress).

What about web-applications? Is that true that for FastCgi or mod_wsgi
deamon mode interpreter and application is started only once per say
100 requests?
--
anatoly t.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib

[Nick Coghlan]

On Thu, Jul 15, 2010 at 4:06 PM, Brett Cannon  wrote:
>> In any case, here my results under a Linux system:
>>
>> $ ./python -m importlib.test.benchmark
>> sys.modules [ 323782 326183 326667 ] best is 326667
>> Built-in module [ 33600 33693 33610 ] best is 33693
>>
>> $ ./python -m importlib.test.benchmark -b
>> sys.modules [ 1297640 1315366 1292283 ] best is 1315366
>> Built-in module [ 58180 57708 58057 ] best is 58180
>
> And this is what might make evaluating importlib tough; while the
> performance is 25% of what it is for import.c, being able to import
> over 300,000 times/second is still damn fast.

Yeah, I think the numbers where the filesystem gets involved are going
to be more relevant. Modules that have already been cached and those
built in to the executable aren't likely to dominate interpreter and
application startup times (which is the main thing I'm worried about
seeing regress).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] importlib

[Brett Cannon]

On Wed, Jul 14, 2010 at 13:01, Antoine Pitrou  wrote:
> On Wed, 14 Jul 2010 12:33:55 -0700
> Brett Cannon  wrote:
>>
>> So I started writing benchmark code in anticipation of needing to prove a
>> minimal performance difference to justify bootstrapping importlib. Right now
>> it only compares importing from sys.modules and built-in modules. You can
>> run it with ``./python.exe -m importlib.test.benchmark``. If you add a `-b`
>> option that will use the built-in __import__ implementation.
>
> In what unit are the numbers?

Imports/second. I'll fix the code to state that.

>
> In any case, here my results under a Linux system:
>
> $ ./python -m importlib.test.benchmark
> sys.modules [ 323782 326183 326667 ] best is 326667
> Built-in module [ 33600 33693 33610 ] best is 33693
>
> $ ./python -m importlib.test.benchmark -b
> sys.modules [ 1297640 1315366 1292283 ] best is 1315366
> Built-in module [ 58180 57708 58057 ] best is 58180

And this is what might make evaluating importlib tough; while the
performance is 25% of what it is for import.c, being able to import
over 300,000 times/second is still damn fast.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] importlib

[Antoine Pitrou]

On Wed, 14 Jul 2010 12:33:55 -0700
Brett Cannon  wrote:
> 
> So I started writing benchmark code in anticipation of needing to prove a
> minimal performance difference to justify bootstrapping importlib. Right now
> it only compares importing from sys.modules and built-in modules. You can
> run it with ``./python.exe -m importlib.test.benchmark``. If you add a `-b`
> option that will use the built-in __import__ implementation.

In what unit are the numbers?

In any case, here my results under a Linux system:

$ ./python -m importlib.test.benchmark 
sys.modules [ 323782 326183 326667 ] best is 326667
Built-in module [ 33600 33693 33610 ] best is 33693

$ ./python -m importlib.test.benchmark -b
sys.modules [ 1297640 1315366 1292283 ] best is 1315366
Built-in module [ 58180 57708 58057 ] best is 58180

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com