[issue25711] Rewrite zipimport from scratch

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Nick Coghlan]

Nick Coghlan  added the comment:

Noticed this was still open when reviewing Elvis's zipimport patch for issue 
34022.

Given the Python implementation has been merged, should we close this as 
resolved, and open new issues for any further changes (performance or 
otherwise)?

--
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset b2984ab9a7c458f8b7ed8978c0c95b109116895d by Serhiy Storchaka in 
branch 'master':
bpo-25711: Remove outdated zipimport tests. (GH-9404)
https://github.com/python/cpython/commit/b2984ab9a7c458f8b7ed8978c0c95b109116895d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Decorater]

Change by Decorater :


--
pull_requests: +8832

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 9da3961f364da2a3ced740230b85ffb4309238d3 by Serhiy Storchaka in 
branch 'master':
bpo-25711: Move _ZipImportResourceReader from importlib to zipimport. (GH-9406)
https://github.com/python/cpython/commit/9da3961f364da2a3ced740230b85ffb4309238d3


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +8828

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +8827

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 79d1c2e6c9d1bc1cf41ec3041801ca1a2b9a995b by Serhiy Storchaka in 
branch 'master':
bpo-25711: Rewrite zipimport in pure Python. (GH-6809)
https://github.com/python/cpython/commit/79d1c2e6c9d1bc1cf41ec3041801ca1a2b9a995b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Brett Cannon]

Brett Cannon  added the comment:

I'm personally in no rush and I assume Serhiy isn't either with 3.8 cut-off 
quite a ways out, so I see no rush.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

Not sure if I'll have time before the core sprints to check the implementation 
with shiv, but I can give it a try then.  Do you mind waiting until then?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Brett Cannon]

Brett Cannon  added the comment:

I think Serhiy's PR is basically done, so now the question is do we want to 
merge it in and drop the C code? ;)

I obviously say yes because this is I/O-bound code and so the switch shouldn't 
be enough of a performance hit to warrant blocking the gain in maintainability 
long-term (especially if we try to clean the module up slowly).

Barry, since I know you work with zip files a lot at work did you want to check 
to make sure perf won't be an issue for your use-case?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Fixed issues on Windows.

--
versions: +Python 3.8 -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Brett Cannon]

Brett Cannon  added the comment:

I'm planning to review the PR at some point.

On Sun, Jul 8, 2018, 00:42 Serhiy Storchaka,  wrote:

>
> Serhiy Storchaka  added the comment:
>
> Could anybody please make a review? This is just a first step, we need to
> do it before making other steps: implementing more modern import API,
> supporting large ZIP files and ZIP files with comments, implementing
> loading binary extensions from ZIP files, etc.
>
> --
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Could anybody please make a review? This is just a first step, we need to do it 
before making other steps: implementing more modern import API, supporting 
large ZIP files and ZIP files with comments, implementing loading binary 
extensions from ZIP files, etc.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[pmpp]

Change by pmpp :


--
nosy: +pmpp

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

PR 6809 freezes zipimport.py, adds changes made in zipimport.c since writing 
the initial Python implementation, and fixes few bugs exposed in the frozen 
module. Seems Python now works with the zipped stdlib.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +6497

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Decorater]

Decorater  added the comment:

Alright, just wanted to make sure because I did not want to have something 
break when loading up the entire standard library from an zip file with it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Brett Cannon]

Brett Cannon  added the comment:

zipimport.py would be frozen just like importlib, so there's no bootstrapping 
issue if that's what you're asking.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Decorater]

Decorater  added the comment:

So, after reviewing this it started to make me rethink about the zipimport.py 
file.

So my question is how would that file work if it is an pyc file in python37.zip 
or something just to zipimport other modules? There is got to be some sort of 
low level api that can zip import the zip importer then on the rewrite. Am I 
right?

Maybe the best bet is to wait for bug reports on the C Code and fixup the C 
Code if possible so that way there is no conflicts like the ones I just 
questioned.

--
nosy: +Decorater

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

I've ported the patch to a branch on GH.  See PR 4023.  I started from 
zipimport-2.patch.  It doesn't work, but it will be easier to work on as a PR 
rather than a patch.

Contributions welcome!  Let's see if we can make this work.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Barry A. Warsaw]

Change by Barry A. Warsaw :


--
pull_requests: +3998

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

I've landed here after chatting with @brett.cannon.  I have a use case for this 
(making pex startup faster by bypassing pkg_resources) but I need to hack 
around the limitation of dlopen'ing .so's from zips.  Our idea was to have a 
zipimport subclass which doesn't return None from `importlib.util.find_spec()` 
when it finds a .so, but instead dumps that into some safe directory, and then 
arranges for a loader that knows how to load that.  It sure would be handy for 
this to be a zipimporter subclass. :)

I think Serhiy's patch predates the move to GitHub, so it's not a branch/PR.  I 
guess the next step would be to branchify the patch and then continue 
discussion over there.  Depending on my availability, I might do that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Barry A. Warsaw]

Change by Barry A. Warsaw :


--
nosy: +barry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Yaron de Leeuw]

Yaron de Leeuw added the comment:

What is the status of this work? Is there anything I can do to help make this 
happen?

--
nosy: +jarondl

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Brett Cannon]

Brett Cannon added the comment:

Just FYI, if this lands I will probably try to build off of it to make a 
pathlib-like zip module to eventually replace zipfile. So if there's any API 
design decisions that need to be made, it would be great if we try to keep the 
zip-specific bits separate and generic enough to work with in other future 
libraries.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Brett Cannon]

Brett Cannon added the comment:

And having a private copy of os.fsdecode() isn't difficult as os.fspath() is in 
posix and after that it's four lines that only need access to the sys module.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

> Technically, will it be possible to freeze it?

I think yes. But I don't know how to do this. I hope on Brett's (or other 
import machinery expert) help.

Since zipimporter constructor is called only with string path by import 
machinery, the os module is not imported at initialization stage.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Gregory P. Smith]

Gregory P. Smith added the comment:

So long as this code block that imports os is avoided, I believe that this can 
be properly frozen:

+if not isinstance(path, str):
+import os
+path = os.fsdecode(path)

But it should be easy to avoid that code path when the standard library is a 
zip file.

Otherwise it uses importlib (frozen), marshal (builtin), sys (builtin), time 
(builtin), and zlib [if present] (extension module).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[STINNER Victor]

STINNER Victor added the comment:

> Here is preliminary translation of zipimport to Python. It is not frozen and 
> imports other modules.

Technically, will it be possible to freeze it? It seems useful to keep the 
ability to put the whole stdlib into a single ZIP. Using a ZIP is sometimes 
suggested to avoid fstat() on disk for example, to speedup Python startup.

But I also understand that the C code is painful to maintain and update.

Anyway, nice job Serhiy :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Got rid of dependencies from os, stat and encodings.

--
Added file: http://bugs.python.org/file45814/zipimport-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Here is preliminary translation of zipimport to Python. It is not frozen and 
imports other modules. I tried to keep the implementation close to C 
implementation. As a consequence, some raised exceptions look arbitrary.

--
keywords: +patch
stage:  -> patch review
type:  -> enhancement
versions: +Python 3.7 -Python 3.6
Added file: http://bugs.python.org/file45800/zipimport.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Rose Ames]

Rose Ames added the comment:

Sorry for the late response.  I didn't have much time over the holidays.  I 
think I better let someone else take this one.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[diana]

Changes by diana :


--
nosy: +diana

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Brett Cannon]

Brett Cannon added the comment:

What are people's statuses on their various attempts? Since this is going to 
block my importlib.resources work I will do the work myself or work directly 
with someone in order to make sure this gets done.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[STINNER Victor]

STINNER Victor added the comment:

Can you both publish your WIP work?

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I were on very early stage, and stopped this work few weeks ago in favor of 
other issues. I would be glad to make a review of your work when you have 
finished it Rose.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Rose Ames]

Rose Ames added the comment:

Serhiy, how far along are you on this?  I have a wip from this summer that I 
could finish over the holidays.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

It was my intention.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I was my intention.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
Removed message: http://bugs.python.org/msg255223

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
Removed message: http://bugs.python.org/msg255222

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

It was my intension.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Eric Snow]

Changes by Eric Snow :


--
nosy: +eric.snow, superluser

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Brett Cannon]

Brett Cannon added the comment:

Are you writing it in such a way that it can be bootstspped in with importlib 
so the stslib can be loaded from it?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

FYI I'm at the early stage of rewriting zipimport in Python.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Robert Byrnes]

Changes by Robert Byrnes :


--
nosy: +byrnes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25711] Rewrite zipimport from scratch

[Brett Cannon]

New submission from Brett Cannon:

No one wants to work on zipimport, and yet it's full of bugs. It needs a 
rewrite so that it's more maintainable. An idea floated at PyCon 2015 was to 
writing the zip-reading code in C and to keep it as simple as possible -- e.g., 
don't worry about supporting comments, etc. -- and then write the rest of the 
code in importlib, making maintenance much easier.

All of the various zipimport bugs should be made dependent on this issue as 
unless they are critical flaws I doubt they will get fixed without the rewrite.

--
components: Interpreter Core
messages: 255183
nosy: brett.cannon, gregory.p.smith, twouters
priority: high
severity: normal
status: open
title: Rewrite zipimport from scratch
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com