[issue1739468] Add a -z interpreter flag to execute a zip file

[Phillip J. Eby]

Phillip J. Eby added the comment:

Patch implementing an alternate approach: support automatically
importing __main__ when sys.argv[0] is an importable path.  This allows
zip files, directories, and any future importable locations (e.g. URLs)
to be used on the command line.  Note that this also means that you
don't need an option on the #! line in a zip file, which avoids hairy #!
issues on platforms like Linux (where a #! line can have at most one
argument).  __main__ is used instead of __zipmain__, since it is not
zipfile specific.

--
nosy: +pje

_
Tracker <[EMAIL PROTECTED]>

_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1739468] Add a -z interpreter flag to execute a zip file

[Phillip J. Eby]

Changes by Phillip J. Eby:


_
Tracker <[EMAIL PROTECTED]>

_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1739468] Add a -z interpreter flag to execute a zip file

[Nick Coghlan]

Nick Coghlan added the comment:

I like PJE's approach, and the patch works for me.

About the only thing I'd change is to switch the expression in
PyImport_GetImporter to a simple chain of if-statements in order to:
  - silence the warning from GCC about an unused value
  - make it more obvious to a reader what the function is doing

An optimising compiler is going to produce similar code either way, and
it took me a moment to realise that the && operations are being used
purely for their short-circuiting effect, even though there is no real
advantage to using an expression instead of a statement at that point in
the code.

Adding a simple test of the functionality to test_cmd_line would also be
good.

_
Tracker <[EMAIL PROTECTED]>

_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1739468] Add a -z interpreter flag to execute a zip file

[Paul Moore]

Paul Moore added the comment:

PJE's patch looks OK. I agree with Nick that the chain of &&s in 
PyImport_GetImporter should be expanded into a chain of ifs. As it 
stands, the code is needlessly obfuscated.

_
Tracker <[EMAIL PROTECTED]>

_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1739468] Add a -z interpreter flag to execute a zip file

[Guido van Rossum]

Guido van Rossum added the comment:

PJE's patch looks good to me too.

Stylistic nits:

- The proper name of the now-public null importer type ought to be
PyNullImporter_Type, to rhyme with e.g. PyString_Type

- There's a multi-line if that has the closing parenthesis in an odd
place at the start of the next line. The preferred style is to place the
close paren after the last condition, and put the open curly on a line
by itself.

--
nosy: +gvanrossum

_
Tracker <[EMAIL PROTECTED]>

_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1739468] Add a -z interpreter flag to execute a zip file

[Nick Coghlan]

Nick Coghlan added the comment:

Attached an updated version of PJE's patch with the suggested cleanups
and a new unit test file (test_cmd_line_script.py). Finding the
roundtuits to finish the latter is actually what has taken me so long.

The basic tests and the directory tests are currently working, but for
some reason the zipfile tests are attempting to load __main__ using
pkgutil.ImpLoader instead of the zipimport module.

I'm posting the patch anyway to see if anyone else can spot where it's
going wrong before I find some more time to try and figure it out for
myself.

Added file: http://bugs.python.org/file8767/runmain_with_tests.diff

_
Tracker <[EMAIL PROTECTED]>

_Index: Python/import.c
===
--- Python/import.c (revision 59036)
+++ Python/import.c (working copy)
@@ -104,7 +104,6 @@
 };
 #endif
 
-static PyTypeObject NullImporterType;  /* Forward reference */
 
 /* Initialize things */
 
@@ -167,7 +166,7 @@
 
/* adding sys.path_hooks and sys.path_importer_cache, setting up
   zipimport */
-   if (PyType_Ready(&NullImporterType) < 0)
+   if (PyType_Ready(&PyNullImporter_Type) < 0)
goto error;
 
if (Py_VerboseFlag)
@@ -1088,7 +1087,7 @@
}
if (importer == NULL) {
importer = PyObject_CallFunctionObjArgs(
-   (PyObject *)&NullImporterType, p, NULL
+   (PyObject *)&PyNullImporter_Type, p, NULL
);
if (importer == NULL) {
if (PyErr_ExceptionMatches(PyExc_ImportError)) {
@@ -1106,6 +1105,20 @@
return importer;
 }
 
+PyAPI_FUNC(PyObject *)
+PyImport_GetImporter(PyObject *path) {
+PyObject *importer=NULL, *path_importer_cache=NULL, *path_hooks=NULL;
+
+   if ((path_importer_cache = PySys_GetObject("path_importer_cache"))) {
+   if ((path_hooks = PySys_GetObject("path_hooks"))) {
+   importer = get_path_importer(path_importer_cache,
+path_hooks, path);
+   }
+   }
+   Py_XINCREF(importer); /* get_path_importer returns a borrowed reference 
*/
+   return importer;
+}
+
 /* Search the path (default sys.path) for a module.  Return the
corresponding filedescr struct, and (via return arguments) the
pathname and an open file.  Return NULL if the module is not found. */
@@ -3049,7 +3062,7 @@
 };
 
 
-static PyTypeObject NullImporterType = {
+PyTypeObject PyNullImporter_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
"imp.NullImporter",/*tp_name*/
sizeof(NullImporter),  /*tp_basicsize*/
@@ -3096,7 +3109,7 @@
 {
PyObject *m, *d;
 
-   if (PyType_Ready(&NullImporterType) < 0)
+   if (PyType_Ready(&PyNullImporter_Type) < 0)
goto failure;
 
m = Py_InitModule4("imp", imp_methods, doc_imp,
@@ -3118,8 +3131,8 @@
if (setint(d, "PY_CODERESOURCE", PY_CODERESOURCE) < 0) goto failure;
if (setint(d, "IMP_HOOK", IMP_HOOK) < 0) goto failure;
 
-   Py_INCREF(&NullImporterType);
-   PyModule_AddObject(m, "NullImporter", (PyObject *)&NullImporterType);
+   Py_INCREF(&PyNullImporter_Type);
+   PyModule_AddObject(m, "NullImporter", (PyObject *)&PyNullImporter_Type);
   failure:
;
 }
Index: Include/import.h
===
--- Include/import.h(revision 59036)
+++ Include/import.h(working copy)
@@ -24,6 +24,7 @@
 #define PyImport_ImportModuleEx(n, g, l, f) \
PyImport_ImportModuleLevel(n, g, l, f, -1)
 
+PyAPI_FUNC(PyObject *) PyImport_GetImporter(PyObject *path);
 PyAPI_FUNC(PyObject *) PyImport_Import(PyObject *name);
 PyAPI_FUNC(PyObject *) PyImport_ReloadModule(PyObject *m);
 PyAPI_FUNC(void) PyImport_Cleanup(void);
@@ -42,6 +43,7 @@
 void (*initfunc)(void);
 };
 
+PyAPI_DATA(PyTypeObject) PyNullImporter_Type;
 PyAPI_DATA(struct _inittab *) PyImport_Inittab;
 
 PyAPI_FUNC(int) PyImport_AppendInittab(char *name, void (*initfunc)(void));
Index: Lib/test/test_cmd_line.py
===
--- Lib/test/test_cmd_line.py   (revision 59036)
+++ Lib/test/test_cmd_line.py   (working copy)
@@ -1,3 +1,6 @@
+# Tests invocation of the interpreter with various command line arguments
+# All tests are executed with environment variables ignored
+# See test_cmd_line_script.py for testing of script execution
 
 import test.test_support, unittest
 import sys
Index: Lib/test/test_cmd_line_script.py
===
--- Lib/test/test_cmd_line_script.py(revision 0)
+++ Lib/test/test_cmd_line_script.py(revision 0)
@@ -0,0 +1,145 @@
+# Tests command line execution of scripts
+from __future__ i

[issue1739468] Add a -z interpreter flag to execute a zip file

[Nick Coghlan]

Nick Coghlan added the comment:

I worked out what was wrong with my unit tests (I was incorrectly
including the path information when adding the test script to the zipfile)

I've updated the patch here, and will be committing the change once the
test suite finishes running.

--
versions: +Python 2.6
Added file: http://bugs.python.org/file8770/runmain_with_tests.diff

_
Tracker <[EMAIL PROTECTED]>

_Index: Python/import.c
===
--- Python/import.c (revision 59036)
+++ Python/import.c (working copy)
@@ -104,7 +104,6 @@
 };
 #endif
 
-static PyTypeObject NullImporterType;  /* Forward reference */
 
 /* Initialize things */
 
@@ -167,7 +166,7 @@
 
/* adding sys.path_hooks and sys.path_importer_cache, setting up
   zipimport */
-   if (PyType_Ready(&NullImporterType) < 0)
+   if (PyType_Ready(&PyNullImporter_Type) < 0)
goto error;
 
if (Py_VerboseFlag)
@@ -1088,7 +1087,7 @@
}
if (importer == NULL) {
importer = PyObject_CallFunctionObjArgs(
-   (PyObject *)&NullImporterType, p, NULL
+   (PyObject *)&PyNullImporter_Type, p, NULL
);
if (importer == NULL) {
if (PyErr_ExceptionMatches(PyExc_ImportError)) {
@@ -1106,6 +1105,20 @@
return importer;
 }
 
+PyAPI_FUNC(PyObject *)
+PyImport_GetImporter(PyObject *path) {
+PyObject *importer=NULL, *path_importer_cache=NULL, *path_hooks=NULL;
+
+   if ((path_importer_cache = PySys_GetObject("path_importer_cache"))) {
+   if ((path_hooks = PySys_GetObject("path_hooks"))) {
+   importer = get_path_importer(path_importer_cache,
+path_hooks, path);
+   }
+   }
+   Py_XINCREF(importer); /* get_path_importer returns a borrowed reference 
*/
+   return importer;
+}
+
 /* Search the path (default sys.path) for a module.  Return the
corresponding filedescr struct, and (via return arguments) the
pathname and an open file.  Return NULL if the module is not found. */
@@ -3049,7 +3062,7 @@
 };
 
 
-static PyTypeObject NullImporterType = {
+PyTypeObject PyNullImporter_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
"imp.NullImporter",/*tp_name*/
sizeof(NullImporter),  /*tp_basicsize*/
@@ -3096,7 +3109,7 @@
 {
PyObject *m, *d;
 
-   if (PyType_Ready(&NullImporterType) < 0)
+   if (PyType_Ready(&PyNullImporter_Type) < 0)
goto failure;
 
m = Py_InitModule4("imp", imp_methods, doc_imp,
@@ -3118,8 +3131,8 @@
if (setint(d, "PY_CODERESOURCE", PY_CODERESOURCE) < 0) goto failure;
if (setint(d, "IMP_HOOK", IMP_HOOK) < 0) goto failure;
 
-   Py_INCREF(&NullImporterType);
-   PyModule_AddObject(m, "NullImporter", (PyObject *)&NullImporterType);
+   Py_INCREF(&PyNullImporter_Type);
+   PyModule_AddObject(m, "NullImporter", (PyObject *)&PyNullImporter_Type);
   failure:
;
 }
Index: Include/import.h
===
--- Include/import.h(revision 59036)
+++ Include/import.h(working copy)
@@ -24,6 +24,7 @@
 #define PyImport_ImportModuleEx(n, g, l, f) \
PyImport_ImportModuleLevel(n, g, l, f, -1)
 
+PyAPI_FUNC(PyObject *) PyImport_GetImporter(PyObject *path);
 PyAPI_FUNC(PyObject *) PyImport_Import(PyObject *name);
 PyAPI_FUNC(PyObject *) PyImport_ReloadModule(PyObject *m);
 PyAPI_FUNC(void) PyImport_Cleanup(void);
@@ -42,6 +43,7 @@
 void (*initfunc)(void);
 };
 
+PyAPI_DATA(PyTypeObject) PyNullImporter_Type;
 PyAPI_DATA(struct _inittab *) PyImport_Inittab;
 
 PyAPI_FUNC(int) PyImport_AppendInittab(char *name, void (*initfunc)(void));
Index: Lib/test/test_cmd_line.py
===
--- Lib/test/test_cmd_line.py   (revision 59036)
+++ Lib/test/test_cmd_line.py   (working copy)
@@ -1,3 +1,6 @@
+# Tests invocation of the interpreter with various command line arguments
+# All tests are executed with environment variables ignored
+# See test_cmd_line_script.py for testing of script execution
 
 import test.test_support, unittest
 import sys
Index: Lib/test/test_cmd_line_script.py
===
--- Lib/test/test_cmd_line_script.py(revision 0)
+++ Lib/test/test_cmd_line_script.py(revision 0)
@@ -0,0 +1,145 @@
+# Tests command line execution of scripts
+from __future__ import with_statement
+
+import unittest
+import os
+import os.path
+import sys
+import test
+import tempfile
+import subprocess
+import py_compile
+import contextlib
+import shutil
+import zipfile
+
+verbose = test.test_support.verbose
+
+# XXX ncoghlan: Should we consid

[issue1739468] Add a -z interpreter flag to execute a zip file

[Nick Coghlan]

Nick Coghlan added the comment:

Committed as rev 59039 (now to see how the buildbots react for other
platforms...)

--
resolution:  -> accepted
status: open -> closed

_
Tracker <[EMAIL PROTECTED]>

_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1739468] Add a -z interpreter flag to execute a zip file

[Nick Coghlan]

Nick Coghlan added the comment:

Reverted status to open until I figure out why the tests are failing on
the Mac OSX buildbot.

--
resolution: accepted -> 
status: closed -> open

_
Tracker <[EMAIL PROTECTED]>

_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1739468] Add a -z interpreter flag to execute a zip file

[Guido van Rossum]

Guido van Rossum added the comment:

I can look into this, as I have OSX on my laptop.

_
Tracker <[EMAIL PROTECTED]>

_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1739468] Add a -z interpreter flag to execute a zip file

[Guido van Rossum]

Guido van Rossum added the comment:

Actually the failures aren't OSX-specific:

==
FAIL: test_directory (__main__.CmdLineTest)
--
Traceback (most recent call last):
  File "Lib/test/test_cmd_line_script.py", line 117, in test_directory
self._check_script(script_dir, script_name, script_dir)
  File "Lib/test/test_cmd_line_script.py", line 96, in _check_script
self.assertEqual(exit_code, 0, data)
AssertionError: /usr/local/google/home/guido/python/py3k/python:
'/tmp/tmpLGqOxc' is a directory, cannot continue


==
FAIL: test_directory_compiled (__main__.CmdLineTest)
--
Traceback (most recent call last):
  File "Lib/test/test_cmd_line_script.py", line 124, in
test_directory_compiled
self._check_script(script_dir, compiled_name, script_dir)
  File "Lib/test/test_cmd_line_script.py", line 96, in _check_script
self.assertEqual(exit_code, 0, data)
AssertionError: /usr/local/google/home/guido/python/py3k/python:
'/tmp/tmprNwPih' is a directory, cannot continue


==
FAIL: test_zipfile (__main__.CmdLineTest)
--
Traceback (most recent call last):
  File "Lib/test/test_cmd_line_script.py", line 130, in test_zipfile
self._check_script(zip_name, None, zip_name)
  File "Lib/test/test_cmd_line_script.py", line 96, in _check_script
self.assertEqual(exit_code, 0, data)
AssertionError:   File "/tmp/tmpInCAJO/test_zip.zip", line 1
PK# statements being executed
  ^
SyntaxError: invalid syntax
[25429 refs]


==
FAIL: test_zipfile_compiled (__main__.CmdLineTest)
--
Traceback (most recent call last):
  File "Lib/test/test_cmd_line_script.py", line 137, in
test_zipfile_compiled
self._check_script(zip_name, None, zip_name)
  File "Lib/test/test_cmd_line_script.py", line 96, in _check_script
self.assertEqual(exit_code, 0, data)
AssertionError:   File "/tmp/tmpqh6g1C/test_zip.zip", line 1
SyntaxError: Non-UTF-8 code starting with '\xc8' in file
/tmp/tmpqh6g1C/test_zip.zip on line 2, but no encoding declared; see
http://python.org/dev/peps/pep-0263/ for details
[25428 refs]

_
Tracker <[EMAIL PROTECTED]>

_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1739468] Add a -z interpreter flag to execute a zip file

[Guido van Rossum]

Guido van Rossum added the comment:

Oops, those are failures under 3.0, probably due to Crys's merge.  On
Linux, the 2.6 version of the test doesn't fail.  I see 2 failing tests
on OSX with the 2.6 version, which I will look into.

_
Tracker <[EMAIL PROTECTED]>

_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1739468] Add a -z interpreter flag to execute a zip file

[Guido van Rossum]

Guido van Rossum added the comment:

Fixed the OSX failure in revision 59055; it was due to /tmp being a
symlink, and fixed by application of realpath().

Keeping this open until the 3.0 version is working.

_
Tracker <[EMAIL PROTECTED]>

_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1739468] Add a -z interpreter flag to execute a zip file

[Guido van Rossum]

Guido van Rossum added the comment:

3.0 fix committed as revision 59058.

--
resolution:  -> accepted
status: open -> closed
versions: +Python 3.0

_
Tracker <[EMAIL PROTECTED]>

_
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com