Re: [Python-Dev] zipimport.c broken with implicit namespace packages

[mike . romberg]

> " " == Brett Cannon  writes:

...

 > So it's possible that some abstraction might be possible, but
 > up to this point there hasn't been a motivating factor for
 > importlib to do this as zipimport is written in C and thus
 > won't benefit from any abstraction that importlib uses in its
 > Python code (hence why zipimport needs to be rewritten). Maybe
 > after we have a zip-backed importer written in Python a common
 > API will fall through and we can abstract that out, but I
 > wouldn't want to guess until the code is written and someone
 > can stare at the two implementations.

  Fair enough.  I too think it is a good idea to make base classes
after their is a need for them and not before.   Some argument could
be made that there is a need now as zipimported modules/packages don't
work exactly the same way as "normal" ones.  But since you plan a
rewrite of zipimport collecting the common stuff after that makes sense.

 > It should also be said that there is nothing requiring that an
 > importer support any concept of a file. Going back to the
 > sqlite database example, you could make it nothing more than a
 > table that maps module name to source code and bytecode:

  Yep.  I was not thinking file either.  I may have said file (again
I'm terrible with names) but I'm thinking an array of bytes with a
string id (which I call a path).  The actual storage could simply be
storing the byte array using the stringID/path.

...

 > That's enough to implement an importer and there is no mention
 > of file paths anywhere. If you really wanted to could do a
 > second table that acted as a database-backed file system for
 > reading package data files, but that is not required to meet
 > the minimum functionality -- and then some thanks to bytecode
 > support -- for an importer.

  Yea the python module name could be viewed as a path
(package.subpackage.module) and stored in a hierarchical way.  Or it
could simply be viewed as a string and stored in some flat database.


  Anyway, back at the ranch...  I've got an extended version of the
patch for issue17633 working on my system.  I've added to the test
case to check for the proper functioning of implicit namespace
packages spread between two zip files.  I still need to add tests for
a namespace package spread between a normal filesystem and a zip
file.  I expect I'll have that done in a day or two.  I'll post it to
the bug tracker.

Mike


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 257 and __init__

[Guido van Rossum]

On Tue, Dec 29, 2015 at 1:03 PM, Facundo Batista 
wrote:

> On Tue, Dec 29, 2015 at 4:38 PM, Andrew Barnert 
> wrote:
>
> > Isn't the same thing true for every special method? There are lots of
> classes where __add___ just says "a.__add__(b) = a + b" or (better
> following the PEP) "Return self + value." But, in the rare case where the
> semantics of "a + b" are a little tricky (think of "a / b" for
> pathlib.Path), where else could you put it but __add__?
> >
> > Similarly, for most classes, there's only one of __init__ or __new__,
> and the construction/initialization semantics are simple enough to describe
> in one line of the class docstring--but when things are more complicated
> and need to be documented, where else would you put it?
>
> Yeap. Note that I'm ok to include a docstring when the actual
> behaviour would deviate from the expected one as per Reference Docs.
> My point is to not make it mandatory.
>
>
> > I usually just don't bother. You can violate PEP 257 when it makes
> sense, just like PEP 8. They're just guidelines, not iron-clad rules.
>
> Yeap, but pep257 (the tool [0]) complains for __init__, and wanted to
> know how serious was it.
>
>
> [0] https://pypi.python.org/pypi/pep257


That is the tool's fault. I personally hate with a vengeance that there are
tools named after style guide PEPs that claim to enforce the guidelines
from those PEPs. The tools' rigidity and simplicity reflects badly on the
PEPs, which try hard not to be rigid or simplistic.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 257 and __init__

[Nick Coghlan]

On 4 January 2016 at 17:01, Terry Reedy  wrote:
> Ask the PSF/pypi people to either prohibit such names or require a
> disclaimer of some sort.  They are inherently confusing: "I took a look at
> pep008" does not mean that one even looked at the PEP.  Even when the
> context makes clear that the referent is the module, there is confusion as
> to its authoritativeness.  That Facudo would post here about the module's
> output illustrates that. To me, the name copying violates our informal
> trademark within Pythonland on 'PEP'.

I don't think that's the right answer, as opinionated tools do serve a
useful purpose in preventing bikeshedding during code review (people
*expect* computers to be annoyingly pedantic, which frees up the human
reviewers to focus on higher level concerns). As projects evolve over
time, they may develop their own tweaks and customisations in their
style guide and switch to a more configurable tool, or they may not.

When some of the default settings for the pep8 utility became a
problem, I was able to talk to the developers and persuade them to
tune their defaults to be more in line with the actual PEP text, and
keep their extensions to optional settings.

A similar approach may work for PEP 257, by clarifying which aspects
tools should be leaving to human judgement (beyond the question of
whether or not to opt in to following PEP 257 at all - it's far less
universal than PEP 8).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 257 and __init__

[Terry Reedy]

On 1/3/2016 6:21 PM, Guido van Rossum wrote:

On Tue, Dec 29, 2015 at 1:03 PM, Facundo Batista
> wrote:

On Tue, Dec 29, 2015 at 4:38 PM, Andrew Barnert > wrote:

> Isn't the same thing true for every special method? There are lots of classes where __add___ just says 
"a.__add__(b) = a + b" or (better following the PEP) "Return self + value." But, in the rare case 
where the semantics of "a + b" are a little tricky (think of "a / b" for pathlib.Path), where else 
could you put it but __add__?
>
> Similarly, for most classes, there's only one of __init__ or __new__, and 
the construction/initialization semantics are simple enough to describe in one 
line of the class docstring--but when things are more complicated and need to be 
documented, where else would you put it?

Yeap. Note that I'm ok to include a docstring when the actual
behaviour would deviate from the expected one as per Reference Docs.
My point is to not make it mandatory.


> I usually just don't bother. You can violate PEP 257 when it makes sense, 
just like PEP 8. They're just guidelines, not iron-clad rules.

Yeap, but pep257 (the tool [0]) complains for __init__, and wanted to
know how serious was it.


[0] https://pypi.python.org/pypi/pep257


That is the tool's fault. I personally hate with a vengeance that there
are tools named after style guide PEPs that claim to enforce the
guidelines from those PEPs. The tools' rigidity and simplicity reflects
badly on the PEPs, which try hard not to be rigid or simplistic.


Ask the PSF/pypi people to either prohibit such names or require a 
disclaimer of some sort.  They are inherently confusing: "I took a look 
at pep008" does not mean that one even looked at the PEP.  Even when the 
context makes clear that the referent is the module, there is confusion 
as to its authoritativeness.  That Facudo would post here about the 
module's output illustrates that. To me, the name copying violates our 
informal trademark within Pythonland on 'PEP'.


--
Terry Jan Reedy


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] zipimport.c broken with implicit namespace packages

[Phil Thompson]

On 3 Jan 2016, at 3:41 am, Guido van Rossum  wrote:
> 
> On Sat, Jan 2, 2016 at 3:26 PM,  wrote:
> 
> --
> > "Brett" == Brett Cannon  writes:
> 
> > I opened
> > https://bugs.python.org/issue25711 to specifically try to
> > fix this issue once and for all and along the way modernize
> > zipimport by rewriting it from scratch to be more
> > maintainable
> 
>   Every time I read about impementing a custom loader:
> 
> https://docs.python.org/3/library/importlib.html
> 
>   I've wondered why python does not have some sort of virtual
> filesystem layer to deal with locating modules/packages/support
> files.   Virtual file systems seem like a good way to store data on a
> wide range of storage devices.
> 
> Yeah, but most devices already implement a *real* filesystem, so the only 
> time the VFS would come in handy would be for zipfiles, where we already have 
> a solution.

Just to point out that it would be nice to have an easier way to use something 
other that zipfiles. I have a need to exploit a different solution and have to 
patch the bootstrap code (because the zipfile support is handled as a special 
case). BTW the need is to create iOS and Android executables from frozen Python 
code.

Phil

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] zipimport.c broken with implicit namespace packages

[Brett Cannon]

On Sun, 3 Jan 2016 at 02:55 Phil Thompson 
wrote:

> On 3 Jan 2016, at 3:41 am, Guido van Rossum  wrote:
> >
> > On Sat, Jan 2, 2016 at 3:26 PM,  wrote:
> >
> > --
> > > "Brett" == Brett Cannon  writes:
> >
> > > I opened
> > > https://bugs.python.org/issue25711 to specifically try to
> > > fix this issue once and for all and along the way modernize
> > > zipimport by rewriting it from scratch to be more
> > > maintainable
> >
> >   Every time I read about impementing a custom loader:
> >
> > https://docs.python.org/3/library/importlib.html
> >
> >   I've wondered why python does not have some sort of virtual
> > filesystem layer to deal with locating modules/packages/support
> > files.   Virtual file systems seem like a good way to store data on a
> > wide range of storage devices.
> >
> > Yeah, but most devices already implement a *real* filesystem, so the
> only time the VFS would come in handy would be for zipfiles, where we
> already have a solution.
>
> Just to point out that it would be nice to have an easier way to use
> something other that zipfiles. I have a need to exploit a different
> solution and have to patch the bootstrap code (because the zipfile support
> is handled as a special case). BTW the need is to create iOS and Android
> executables from frozen Python code.
>

Not quite sure about how zip files are a special-case beyond just being put
in sys.meta_path automatically. You can get the same results with a .pth
file or a sitecustomize.py depending on how pervasive your need is.
Otherwise feel free to file an issue at bugs.python.org and we can talk
over there about what you specifically need and if it's reasonable to try
and support.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] zipimport.c broken with implicit namespace packages

[Brett Cannon]

On Sat, 2 Jan 2016 at 21:31  wrote:

> > " " == Brett Cannon  writes:
>
>  > I just wanted to quickly say that Guido's observation as to how
>  > a VFS is overkill is right. Imagine implementing a loader using
>  > sqlite and you quickly realize that doing a dull VFS is more
>  > than necessary to implement what import needs to function.
>
>   I fear I've made a poor choice in calling this abstract class a VFS
> (I'm terrible with names).  I'm not thinking of anything along the
> lines of a full file system that supports open(), seek(), read() and
> everything else.   That for sure would be overkill and way more
> complicated than it needs to be.
>
>   All I'm really thinking about is a simple abstract interface that is
> used by an importer to actually locate and retrieve the binary objects
> that will be loaded.  For the simple case I think just two methods
> would/could server a read only "blob/byte database":
>
>   listdir(path)  # returns an iterable container of "files"/"dirs" found
>  # at path
>
>   get(path)  # returns a bytes object for the given path
>
>   I think with those two virtual calls a more high level import layer
> can locate and retrieve modules to be loaded without even knowing
> where they came from.
>
>   The higher level would know about things such as the difference
> between .py and .pyc "files" or the possible existance of __pycache__
> directories and what may be found in them.  Right now the zipimporter
> contains a list of file extensions to try and load (and in what
> order).  It also lacks any knowledge of __pycache__ directories (which
> is one of the outstanding bugs with it).   It just seems to me that
> this sorta logic would be better moved to a higher layer and the zip
> layer just translates paths into reads of byte blobs.
>
>   I mentioned write()/put() for two reasons:
>
>   1)  When I import a .py file then a .pyc file is created on my
>   filesystem.  I don't really know what piece of code created it.
>   But a write to the filesystem (assuming it is writeable and
>   permissions set etc) occurs.   It might be nice for other
>   storage systems (zip, sql, etc) could optionally support this.
>   They could if the code that crated the .pyc simply did a put()
>   to the object that pulled in the .py file.  The interface is
>   expanded by two calls (put() and delete()).
>
>   2)  Integration with package data.  I know there are
>   modules/packages out there that help a module try and locate
>   data files that may be associated with a package.  I think it
>   would be kinda cool for a module to instead be able to get a
>   handle to the abstract class that loaded it.  it could then use
>   the same listdir() get() and possibly write methods the importer
>   did.  The writing bit of this may or may not be a good idea :)
>
>
So it's possible that some abstraction might be possible, but up to this
point there hasn't been a motivating factor for importlib to do this as
zipimport is written in C and thus won't benefit from any abstraction that
importlib uses in its Python code (hence why zipimport needs to be
rewritten). Maybe after we have a zip-backed importer written in Python a
common API will fall through and we can abstract that out, but I wouldn't
want to guess until the code is written and someone can stare at the two
implementations.

It should also be said that there is nothing requiring that an importer
support any concept of a file. Going back to the sqlite database example,
you could make it nothing more than a table that maps module name to source
code and bytecode:

CREATE TABLE code (
name TEXT PRIMARY KEY NOT NULL,
source BLOB,
bytecode BLOB
);
/* A trigger that re-generates `bytecode` when `source` is set would be
nice in this example, but that's beyond my SQL abilities. */

That's enough to implement an importer and there is no mention of file
paths anywhere. If you really wanted to could do a second table that acted
as a database-backed file system for reading package data files, but that
is not required to meet the minimum functionality -- and then some thanks
to bytecode support -- for an importer.


>
>   Anyway, hope I did not muddy the waters.  I was just thinking a bit
> out loud and none of this may live past my own experiments.   I was/am
> just trying to think of why the importers like the zipimporter don't
> work like a filesystem importer and how they would be cleaner if they
> just dealt with paths and byte blobs to store/get based on those paths.
>

It's a reasonable thing to consider, but it would be better to get
zipimport fixed for you, then rewritten, and then look at code for the
rewritten zipimport and what importlib.machinery.SourceFileLoader has to
see if there is a common API to abstract out and build upon.
___
Python-Dev mailing list