Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
I'd have expect this test to fail on _any_ UNIX system if run as root. Root's allowed to write to stuff! Any stuff! About the only permission with any effect on root is the eXecute bit for the exec call, to prevent blindly running random data files. You're right, here's another test on Linux (I must have screwed up when I tested this on my box): # mkdir /tmp/foo # chmod -w /tmp/foo # touch /tmp/foo/bar # ls /tmp/foo bar You can still set the directory immutable if you really want to deny write to root: # chattr +i /tmp/foo # touch /tmp/foo/spam touch: cannot touch `/tmp/foo/spam': Permission denied Equally, why on earth are you running tests as root!?!?!?!?! Madness. It's as bad as compiling stuff as root etc etc. A bad idea all around, securitywise. Agreed, I would personally never run a buildbot as root. I just changed this because I was tired of seeing the same buildbots always red (thus masking real failures). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module
That's not a given. Depending on the memory allocator, a copy can be avoided. That's why the str += str hack is much more efficient under Linux than Windows, AFAIK. Even Linux will have to copy a block on realloc in certain cases, no? Probably so. How often is totally unknown to me :) http://www.gnu.org/software/libc/manual/html_node/Changing-Block-Size.html It depends on whether there's enough free memory after the buffer you currently have allocated. I suppose that this becomes a question of what people consider the general case :-) But under certain circumstances (if a large block is requested), the allocator uses mmap(), no? That's right, if the block requested is bigger than mmap_threshold (256K by default with glibc, forgetting the sliding window algorithm): I'm not sure of what percentage of strings/buffers are concerned in a typical program. In which case mremap() should allow to resize without copying anything. Yes, there's no copying. Note however that it doesn't come for free, the kernel will still zero-fill the pages before handling them to user-space. It is still way faster than on, let's say, Solaris. cf ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using PEP384 Stable ABI for the lzma extension module
Le 6 octobre 2011 10:09, Charles-François Natali neolo...@free.fr a écrit : But under certain circumstances (if a large block is requested), the allocator uses mmap(), no? That's right, if the block requested is bigger than mmap_threshold (256K by default with glibc, forgetting the sliding window algorithm): I'm not sure of what percentage of strings/buffers are concerned in a typical program. Most usages of _PyBytes_Resize() are in compression libraries. 256K payloads are not rare in this area. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
On Oct 5, 2011, at 10:46 PM, Cameron Simpson wrote: Surely VERY FEW tests need to be run as root, and they need careful consideration. The whole thing (build, full test suite) should not run as root. This is news to me - is most of Python not supported to run as root? I was under the impression that Python was supposed to run correctly as root, and therefore there should be some buildbots dedicated to running it that way. If only a few small parts of the API are supposed to work perhaps this should be advertised more clearly in the documentation? Ahem. Sorry for the snark, I couldn't resist. As terry more reasonably put it: running buildbot tests as root does not reflect the experience of non-root users. It seems some tests need to be run both ways just for correctness testing. (except I'd say all, not some) Am I really the only person who feels unease about this scenario? More seriously: apparently you are not, but I am quite surprised by that revelation. You should be :). The idea of root as a special, magical place where real ultimate power resides is quite silly. root is a title, like king. You're not just root, you're root _of_ something. If the thing that you are root of is a dedicated virtual machine with no interesting data besides the code under test, then this is quite a lot like being a regular user in a similarly boring place. It's like having the keys to an empty safe. Similarly, if you're a normal unprivileged user - let's say, www-data - on a system with large amounts of sensitive data owned by that user, becoming root will rarely grant you any really interesting privileges beyond what you've already got. Most public web-based systems fall into this category, as you've got one user (the application deployment user) running almost all of your code, with privileges to read and write to the only interesting data source (the database). So if these tests were running on somebody's public-facing production system in an unprivileged context, I'd be far more concerned about that than about it having root on some throwaway VM. -glyph ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New stringbench benchmark results
Hum, copy-paste failure, I wrote numbers in the wrong order, it's: (test: Python 3.2 = Python 3.3) A.join([Bob]*100)): 0.92 = 2.11 (C+AB*300).rfind(CA): 0.57 = 1.03 (A + (Z*128*1024)).replace(A, BB, 1): 0.25 = 0.50 I improved str.replace(): it's now 5 times faster instead of 2 times slower for this specific benchmark :-) (or 10 times faster in Python 3.3 before/after my patch) Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] check for PyUnicode_READY look backwards
Hi, with the new Unicode API, there are many checks like: +if (PyUnicode_READY(*filename)) +goto handle_error; Every time I read it, I get it wrong: If filename is ready, then fail then I have to remember that the function returns either 0 or -1. I'd prefer it was written : if (PyUnicode_READY(*filename) 0) because 0 clearly indicates an error condition. That's how all calls to PyType_Ready are written, for example. Am I the only one to be distracted by this idiom? -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: PyUnicode_FromKindAndData() raises a ValueError if the kind is unknown
Hi, Le 03/10/2011 23:38, Terry Reedy a écrit : Is it both technically possible (with hg) and socially permissible (with us) to edit another's commit message? Not easily. A changeset identifier is a hash of date, user, parent changesets hashes, commit message and diff content; editing the commit message makes a new changeset. I’ve read about a company where they use a script or an extension to send changesets to a colleague for review and destroy them locally, so that when they pull the changeset edited by the reviewer, they don’t get duplicates. It sounds complicated. Regards ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: PyUnicode_FromKindAndData() raises a ValueError if the kind is unknown
Hi, Le 04/10/2011 04:59, Stephen J. Turnbull a écrit : Currently, in hg. git has a mechanism for adding notes which are automatically displayed along with the original commit message, and bzr is considering introducing such a mechanism. Mercurial commits can contain an “extra” dictionary, but that feature is not yet exposed on the command line. I'm not familiar with the hg dev process (I use hg a lot, but so far it Just Works for me :), but I would imagine they will move in that direction as well. I doubt it; Mercurial has a very strong view of “history is sacred”; it has taken many, many requests for an optional rebase feature to be added, for example. Regards ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: When expandtabs() would be a no-op, don't create a duplicate string
Hi, http://hg.python.org/cpython/rev/447f521ac6d9 user:Antoine Pitrou solip...@pitrou.net date:Tue Oct 04 16:04:01 2011 +0200 summary: When expandtabs() would be a no-op, don't create a duplicate string diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -1585,6 +1585,10 @@ return self.assertRaises(OverflowError, 't\tt\t'.expandtabs, sys.maxsize) +def test_expandtabs_optimization(self): +s = 'abc' +self.assertIs(s.expandtabs(), s) Shouldn’t that be marked CPython-specific? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Rename PyUnicode_KIND_SIZE ?
Hello, The PyUnicode_KIND_SIZE macro is defined as follows. Its name looks rather mysterious or misleading to me. Could we rename it to something else? (also, is it useful? PEP 393 has added a flurry of new macros to unicodeobject.h and it's getting hard to know which ones are genuinely useful, or well-performing) /* Compute (index * char_size) where char_size is 2 ** (kind - 1). The index is a character index, the result is a size in bytes. See also PyUnicode_CHARACTER_SIZE(). */ #define PyUnicode_KIND_SIZE(kind, index) \ ((Py_ssize_t) ((index) ((kind) - 1))) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Status of the built-in virtualenv functionality in 3.3
Hi, I started to play with virtualenv recently and wondered about the status of the similar feature in 3.3 (cpythonv). The last thread mentioned two bugs; one has been fixed since. Apart from the implicit vs. explicit download of distribute, are there design issues to discuss? Can we do that with a patch on a bug report? Oh, let’s not forget naming. We can’t reuse the module name virtualenv as it would shadow the third-party module name, and I’m not fond of “virtualize”: it brings OS-level virtualization to my mind, not isolated Python environments. Cheers ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Fix find_module_path(): make the string ready
+if (PyUnicode_READY(path_unicode)) +return -1; + I think we need to discuss/reconsider the return value of PyUnicode_READY. It's defined to give -1 on error currently. If that sounds good, then the check for error should be a check that it is -1. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
Am 06.10.2011 16:12, schrieb Éric Araujo: Oh, let’s not forget naming. We can’t reuse the module name virtualenv as it would shadow the third-party module name, and I’m not fond of “virtualize”: it brings OS-level virtualization to my mind, not isolated Python environments. How about clutch? A virtualenv is a clutch of Python eggs, all ready to hatch. (Pun intended). :) Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
On Thu, Oct 6, 2011 at 09:12, Éric Araujo mer...@netwok.org wrote: Oh, let’s not forget naming. We can’t reuse the module name virtualenv as it would shadow the third-party module name, and I’m not fond of “virtualize”: it brings OS-level virtualization to my mind, not isolated Python environments. How about we just drop the virtual part of the name and make it env? (or something non-virtual) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] check for PyUnicode_READY look backwards
On 6 Oct, 2011, at 14:57, Amaury Forgeot d'Arc wrote: Hi, with the new Unicode API, there are many checks like: +if (PyUnicode_READY(*filename)) +goto handle_error; Every time I read it, I get it wrong: If filename is ready, then fail then I have to remember that the function returns either 0 or -1. I'd prefer it was written : if (PyUnicode_READY(*filename) 0) because 0 clearly indicates an error condition. That's how all calls to PyType_Ready are written, for example. Am I the only one to be distracted by this idiom? I prefer the ' 0' variant as well, for the same reason as you. Ronald -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com smime.p7s Description: S/MIME cryptographic signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
On Oct 06, 2011, at 04:12 PM, Éric Araujo wrote: I started to play with virtualenv recently and wondered about the status of the similar feature in 3.3 (cpythonv). The last thread mentioned two bugs; one has been fixed since. Apart from the implicit vs. explicit download of distribute, are there design issues to discuss? Can we do that with a patch on a bug report? Oh, let’s not forget naming. We can’t reuse the module name virtualenv as it would shadow the third-party module name, and I’m not fond of “virtualize”: it brings OS-level virtualization to my mind, not isolated Python environments. Time to hit the hardware store and stock up on bikeshed paint! I agree we can't use virtualenv, and shouldn't use virtualize. I'm afraid that picking something cute might make it harder to discover. `pythonv` or `cpythonv` seem like good choices to me. Maybe the former, so we could potentially have jythonv, etc. -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
On Thu, 6 Oct 2011 10:06:17 -0500 Brian Curtin brian.cur...@gmail.com wrote: On Thu, Oct 6, 2011 at 09:12, Éric Araujo mer...@netwok.org wrote: Oh, let’s not forget naming. We can’t reuse the module name virtualenv as it would shadow the third-party module name, and I’m not fond of “virtualize”: it brings OS-level virtualization to my mind, not isolated Python environments. How about we just drop the virtual part of the name and make it env? pythonenv? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
Le 06/10/2011 17:31, Barry Warsaw a écrit : I agree we can't use virtualenv, and shouldn't use virtualize. I'm afraid that picking something cute might make it harder to discover. `pythonv` or `cpythonv` seem like good choices to me. Maybe the former, so we could potentially have jythonv, etc. I’m not sure we would. The feature is two-fold: - changes to getpath.c, site.py and other usual suspects so that CPython supports being run in an isolated environment; - a new module used to create isolated environments. The first part is implemented in CPython; the second part needs a module name to replace virtualenv. python -m pythonv doesn’t seem right. python -m makeenv? python -m workon? (idea from virtualenvwrapper) python -m nest? Cheers ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] counterintuitive behavior (bug?) in Counter with +=
Lars Buitinck wrote: from collections import Counter a = Counter([1,2,3]) b = a a += Counter([3,4,5]) a is b False would become # snip a is b True Sounds like a good idea to me. You should open an issue in the tracker at http://bugs.python.org/. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Rename PyUnicode_KIND_SIZE ?
Le 06/10/2011 15:52, Antoine Pitrou a écrit : The PyUnicode_KIND_SIZE macro is defined as follows. Its name looks rather mysterious or misleading to me. Could we rename it to something else? What do you propose? also, is it useful? index (kind - 1) and index * PyUnicode_CHARACTER_SIZE(str) were used in unicodeobject.c. It's not easy to understand this formula, and it heavily depend on how kind is defined. I wrote a patch to use enum for kind using different values, but the gain was minor so I didn't commit it. We may move it to unicodeobject.c. index * PyUnicode_CHARACTER_SIZE(str) is enough for the public API. (PyUnicode_KIND_SIZE() is also a micro-optimization, it uses shift instead of multiply.) PEP 393 has added a flurry of new macros to unicodeobject.h and it's getting hard to know which ones are genuinely useful, or well-performing) Yes, we have to review new functions and macros. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
On Thu, Oct 6, 2011 at 10:46, Éric Araujo mer...@netwok.org wrote: Le 06/10/2011 17:31, Barry Warsaw a écrit : I agree we can't use virtualenv, and shouldn't use virtualize. I'm afraid that picking something cute might make it harder to discover. `pythonv` or `cpythonv` seem like good choices to me. Maybe the former, so we could potentially have jythonv, etc. I’m not sure we would. The feature is two-fold: - changes to getpath.c, site.py and other usual suspects so that CPython supports being run in an isolated environment; - a new module used to create isolated environments. The first part is implemented in CPython; the second part needs a module name to replace virtualenv. python -m pythonv doesn’t seem right. python -m makeenv? python -m workon? (idea from virtualenvwrapper) python -m nest? develop? devenv? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
On Oct 06, 2011, at 05:46 PM, Éric Araujo wrote: Le 06/10/2011 17:31, Barry Warsaw a écrit : I agree we can't use virtualenv, and shouldn't use virtualize. I'm afraid that picking something cute might make it harder to discover. `pythonv` or `cpythonv` seem like good choices to me. Maybe the former, so we could potentially have jythonv, etc. I’m not sure we would. The feature is two-fold: - changes to getpath.c, site.py and other usual suspects so that CPython supports being run in an isolated environment; - a new module used to create isolated environments. While the other implementations might not be able to share any of CPython's code, it's still a worthy feature for any Python implementation I think. The first part is implemented in CPython; the second part needs a module name to replace virtualenv. python -m pythonv doesn’t seem right. Nope, although `python -m virtualize` seems about perfect. I don't particularly like the -m interface though. Yes, it should work, but I also think there should be a command that basically wraps whatever the -m invocation is, just for user friendliness. python -m makeenv? python -m workon? (idea from virtualenvwrapper) python -m nest? Well, I have to be honest, I've *always* thought nest would be a good choice for a feature like this, but years ago (IIRC) PJE wanted to reserve that term for something else, which I'm not sure ever happened. There's a PyNEST project here: http://www.nest-initiative.uni-freiburg.de/index.php/PyNEST which might cause problems with a built-in `nest` module. Still, I'm a bit fond of `python -m nest` and a `pynest` wrapper. Barring that, `python -m virtualize` with an appropriate cli shortcut (`pysolate`? - say it out loud :) seems good. -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Rename PyUnicode_KIND_SIZE ?
On Thu, 06 Oct 2011 17:52:05 +0200 Victor Stinner victor.stin...@haypocalc.com wrote: index (kind - 1) and index * PyUnicode_CHARACTER_SIZE(str) were used in unicodeobject.c. It's not easy to understand this formula index * PyUnicode_CHARACTER_SIZE(str) is quite easy to understand to me. I find it less cryptic than PyUnicode_KIND_SIZE(kind, index), actually, and I would advocate using the former and removing the latter. (PyUnicode_KIND_SIZE() is also a micro-optimization, it uses shift instead of multiply.) I don't know, but I think the compiler should be able to do that for you. Also, I don't think PyUnicode_KIND_SIZE would be used in a critical loop. You would use PyUnicode_READ when doing one-character-at-a-time stuff. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
On Thu, 6 Oct 2011 12:02:05 -0400 Barry Warsaw ba...@python.org wrote: The first part is implemented in CPython; the second part needs a module name to replace virtualenv. python -m pythonv doesn’t seem right. Nope, although `python -m virtualize` seems about perfect. `python -m sandbox` ? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Éric, Vinay is more up to date than I am on the current status of the implementation. I need to update the PEP draft we worked on last spring and get it posted (the WIP is at https://bitbucket.org/carljm/pythonv-pep but is out of date with the latest implementation work). On 10/06/2011 08:12 AM, Éric Araujo wrote: Oh, let’s not forget naming. We can’t reuse the module name virtualenv as it would shadow the third-party module name, and I’m not fond of “virtualize”: it brings OS-level virtualization to my mind, not isolated Python environments. What about venv? It's short, it's already commonly used colloquially to refer to virtualenv so it makes an accurate and unambiguous mental association, but AFAIK it is currently unused as a script or module name. Carl -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6N0+QACgkQ8W4rlRKtE2fCOwCg1YOWcMCZH6HOdyKepcQG3RgB T48AoIIqol+sUpOAFI+4HJH/dAdX5Xwm =DLjq -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
- Original Message -
I started to play with virtualenv recently and wondered about the status
of the similar feature in 3.3 (cpythonv). The last thread mentioned two
bugs; one has been fixed since.
The pythonv branch is pretty much up to date with the default branch (3.3). I
regularly merge with default and post the results of running the Python
regression suite in a virtual environment - in fact I'm running such a test
right now :-). The test results and a screencast are linked from the project's
BitBucket page at
https://bitbucket.org/vinay.sajip/pythonv/
Apart from the implicit vs. explicit download of distribute, are there
design issues to discuss? Can we do that with a patch on a bug report?
I've made changes to get packaging to work well with virtualenv, some of which
I raised as packaging issues on the tracker. In some cases, I've fixed them in
the pythonv branch. The last bug is a problem with test_user_similar which has
an existing issue (#9100) in a non-virtual environment. This is not a
show-stopper, but I'm not really sure how the user scheme is supposed to work
in venvs: perhaps Carl has a view on this. BTW there have been intermittent
failures in test_packaging, too but they've generally been fixed by changes in
core).
In terms of design issues, it would be useful if someone (apart from me that
is) could look at how the pythonv fork differs from the core and comment on any
issues they find. (Much of the change can be summed up as replacing occurrences
of sys.prefix with sys.getattr('site_prefix', sys.prefix).)
BitBucket makes this type of comparison fairly easy to do; I'm not sure if
there's a lot of value in adding a patch on the tracker for Rietveld review,
until (and if) the PEP is accepted.
Re. distribute: At the moment the pythonv branch downloads a private version of
distribute. The only significant difference from the vanilla distribute is the
use of sys.site_prefix and sys.site_exec_prefix (falling back to sys.prefix and
sys.exec_prefix if we're not running distribute in a virtual env, so it's
backward compatible - but I didn't ask anyone in the packaging/distribute team
to port this small change across. The only reference to sys.site_prefix is this
code in setuptools/command/easy_install.py:
if hasattr(sys, 'site_prefix'):
prefixes = [sys.site_prefix]
else:
prefixes = [sys.prefix]
if sys.exec_prefix != sys.prefix:
prefixes.append(sys.exec_prefix)
If this were ported to distribute, that would be nice :-)
I think the plan is to remove the distribute-downloading functionality from the
stdlib. However, I am working on a companion project, nemo, which will have
this functionality and in addition provides virtualenvwrapper-like
functionality for Linux, Mac and Windows. (This is based on the stdlib API, is
WIP, not released yet, though shown in the screencast I mentioned earlier).
Oh, let’s not forget naming. We can’t reuse the module name virtualenv
as it would shadow the third-party module name, and I’m not fond of
“virtualize”: it brings OS-level virtualization to my mind, not isolated
Python environments.
I'm OK with Carl's suggestion of venv, and prefer it to Brian's suggestion of
env.
Regards,
Vinay Sajip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
On Oct 06, 2011, at 06:04 PM, Antoine Pitrou wrote: On Thu, 6 Oct 2011 12:02:05 -0400 Barry Warsaw ba...@python.org wrote: The first part is implemented in CPython; the second part needs a module name to replace virtualenv. python -m pythonv doesn’t seem right. Nope, although `python -m virtualize` seems about perfect. `python -m sandbox` ? That's nice too. -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
+1 for env or sandbox or something else with box in it. pythonbox? envbox? boxenv? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
On Thu, Oct 6, 2011 at 12:02 PM, Barry Warsaw ba...@python.org wrote: Well, I have to be honest, I've *always* thought nest would be a good choice for a feature like this, but years ago (IIRC) PJE wanted to reserve that term for something else, which I'm not sure ever happened. Actually, it was pretty much for this exact purpose -- i.e. it was the idea of a virtual environment. Ian just implemented it first, with some different ideas about configuration and activation. Since this is basically the replacement for that, I don't have any objection to using the term here. (In my vision, nest was also the name of a package management tool for creating such nests and manipulating their contents, though.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
On 10/06/11 18:02, Barry Warsaw wrote: On Oct 06, 2011, at 05:46 PM, Éric Araujo wrote: Le 06/10/2011 17:31, Barry Warsaw a écrit : I agree we can't use virtualenv, and shouldn't use virtualize. I'm afraid that picking something cute might make it harder to discover. `pythonv` or `cpythonv` seem like good choices to me. Maybe the former, so we could potentially have jythonv, etc. I’m not sure we would. The feature is two-fold: - changes to getpath.c, site.py and other usual suspects so that CPython supports being run in an isolated environment; - a new module used to create isolated environments. While the other implementations might not be able to share any of CPython's code, it's still a worthy feature for any Python implementation I think. The first part is implemented in CPython; the second part needs a module name to replace virtualenv. python -m pythonv doesn’t seem right. Nope, although `python -m virtualize` seems about perfect. Hmm, with proper interpreter support I don't see what would be so virtual about it anymore. Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
On 6 October 2011 17:02, Barry Warsaw ba...@python.org wrote: I don't particularly like the -m interface though. Yes, it should work, but I also think there should be a command that basically wraps whatever the -m invocation is, just for user friendliness. No problem with a wrapper, but the nice thing about the -m form is that it's portable. On Unix, shell script wrappers are pretty portable (no idea if C-shell users would agree...) On Windows, though, there are all sorts of problems. BAT files don't nest, so you end up having to use atrocities like CALL virtualenv within a batch file. Powershell users prefer .ps1 files. The only common form is an EXE, but nobody really likes having to use a compiled form every time. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
Éric Araujo merwok at netwok.org writes: Oh, let’s not forget naming. We can’t reuse the module name virtualenv as it would shadow the third-party module name, and I’m not fond of “virtualize”: it brings OS-level virtualization to my mind, not isolated Python environments. Another possible name would be isolate: python -m isolate /project/env doesn't look too bad. There's no eponymous package on PyPI, and note also that in addition to the common usage of isolate as a verb, it's also a noun with an appropriate meaning in this context. Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] check for PyUnicode_READY look backwards
On Thu, Oct 6, 2011 at 10:31 AM, Ronald Oussoren ronaldousso...@mac.com wrote: On 6 Oct, 2011, at 14:57, Amaury Forgeot d'Arc wrote: I'd prefer it was written : if (PyUnicode_READY(*filename) 0) because 0 clearly indicates an error condition. That's how all calls to PyType_Ready are written, for example. Am I the only one to be distracted by this idiom? I prefer the ' 0' variant as well, for the same reason as you. +1 here as well. The Unix/C 0 as success idiom breaks my Python conditioned brain, so including the explicit 0 in the C code helps resolve that impedance mismatch. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
On Thu, Oct 6, 2011 at 12:50 PM, Barry Warsaw ba...@python.org wrote: On Oct 06, 2011, at 06:04 PM, Antoine Pitrou wrote: On Thu, 6 Oct 2011 12:02:05 -0400 Barry Warsaw ba...@python.org wrote: The first part is implemented in CPython; the second part needs a module name to replace virtualenv. python -m pythonv doesn’t seem right. Nope, although `python -m virtualize` seems about perfect. `python -m sandbox` ? That's nice too. sandbox is a bit close to Victor's pysandbox for restricted execution environments. 'nest' would probably work, although I don't recall the 'egg' nomenclature featuring heavily in the current zipimport or packaging docs, so it may be a little obscure. 'pyenv' is another possible colour for the shed, although a quick Google search suggests that may have few name clash problems. 'appenv' would be yet another colour, since that focuses on the idea of 'environment per application'. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: PyUnicode_FromKindAndData() raises a ValueError if the kind is unknown
Éric Araujo writes: Le 04/10/2011 04:59, Stephen J. Turnbull a écrit : I'm not familiar with the hg dev process (I use hg a lot, but so far it Just Works for me :), but I would imagine they will move in that direction as well. That direction being ability to attach notes to existing commits. It is technically possible to design a VCS in which log messages are ahistorical, and therefore editable at will. I think this would socially be a disaster (though I don't have strong evidence for that view, the fact that the three most popular systems all implement log messages as part of history is very suggestive that some important consideration is involved). The alternative is attaching notes, which are automatically displayed by the VCS's history-viewing tools. I doubt it; Mercurial has a very strong view of “history is sacred”; it has taken many, many requests for an optional rebase feature to be added, for example. While such cultists may prefer Mercurial to git because of the former's conservative position on features like rebase, the devs are apparently pragmatic about editing the DAG. On the one hand, it is now well-understood that rebase is very dangerous. Eg, *nobody* has blanket permission to push rebased branches to Linus's repos, and exceptions are rarely if ever granted. Many reasons for restricting rebase are technical, and AFAICS the Mercurial devs initially took the conservative position that it's more trouble than it's worth. They were forced to change their minds. On the other hand, the Mercurial queues feature is nothing more nor less than history manipulation (in fact, its history manipulations are theoretically equivalent to those of rebase), but in the form of mq it has always been considered acceptable. The reason that mq is acceptable while rebase is not is simply that *references* to the parts of history maintained by mq are not propagated by Mercurial itself (except that now there is a limited ability to do so, but still very restricted and only on explicit request). A notes feature should therefore be acceptable, since it doesn't involve any dangerous changing of refs; the only issue is designing the UI. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/06/2011 12:12 PM, Nick Coghlan wrote: sandbox is a bit close to Victor's pysandbox for restricted execution environments. 'nest' would probably work, although I don't recall the 'egg' nomenclature featuring heavily in the current zipimport or packaging docs, so it may be a little obscure. 'pyenv' is another possible colour for the shed, although a quick Google search suggests that may have few name clash problems. 'appenv' would be yet another colour, since that focuses on the idea of 'environment per application'. I still think 'venv' is preferable to any of the other options proposed thus far. It makes the virtualenv ancestry clearer, doesn't repeat py (which seems entirely unnecessary in the name of a stdlib module, though it could be prepended to a script name if we do a script), and doesn't try to introduce new semantic baggage to the concept, which is already familiar to most Python devs under the name virtualenv. Carl -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6N9H4ACgkQ8W4rlRKtE2doVACcChCim7CNS0czZisjEmw9NblS MqkAn1FyT+A/UiKodCh1siHrQXf2/yZQ =TAUV -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On Oct 06, 2011, at 12:33 PM, Carl Meyer wrote: I still think 'venv' is preferable to any of the other options proposed thus far. It's also nicely unique for googling. Funnily enough, the top hit right now for 'venv' is apparently Lua's project of the same name for the same purpose. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQIcBAEBCAAGBQJOjfwcAAoJEBJutWOnSwa/JRwQALLgtxOAmI9dhgaxudZgsDS7 ZeyaKFTDp2gnteMwXPFv2KK0n1SCsCIzOj807xel8id2Mr5R4FeoBC/HmySSsbxv 2U29QxXHAl/a3pZ50C5a3O7ZV4DShXuPQC/Y430PkwAkrP1ydb/I7KQEgmvHe2A1 UJQ1zbdX+gadk6n9sUo7NuEtvL7lMQdDeGLXvflqRJrVLeGxYV3AHwjByBE6MggU n5tZNv5G/H4fr/pD0SoMlI7MKoWRJkdqAeo3ASySVDn8LXe9UNjJb21YW18RyZPv qcdTzD4NYLJy5FQCmG9N2mOGwUjvrt82kLkcoKcNBekY5TfELshlqgqn31n+bxPo yV4Mr2IFOpRFTW218hwPbpEGK6Mfe03AV59Qey0zSKOsiJPUiPhXu9XdozvlF+Bx 7OdvhTe2nQBf8lp/KaKj4uZnnAvA9C8mMoukn6ly0Kk0EcDw81Ls9nMYxg/3gH7b 0SfvJH+8uQBjXF24Ce/xQkm6D7cB5e9GeilruH9VZ4LfZXYpP7jMdvWhW0g2S4PD KJWChqetOajJBZmWWK0vSBuLAjVxJTZ0Y5q0k9BNOiFj8/5v3QvpCAXQFFv7LTcX RDPI8rk73qjZiyAsVHMOmjSZfpY3aJnhPVMBSn0++yCzWx+YQbPwWkt6VihC6Ve4 Od0WX6XSEEu5BxJYaM/v =/f38 -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What it takes to change a single keyword.
On 10/06/2011 12:00 AM, Brett Cannon wrote: Please file a bug about the dead links so we can fix/remove them. it's done in http://bugs.python.org/issue13117 (and I've also tried with a patch). Cheers, francis ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
Lennart Regebro wrote: +1 for env or sandbox or something else with box in it. Eggbox? Eggcrate? Incubator? -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
On Thu, Oct 6, 2011 at 3:06 PM, Barry Warsaw ba...@python.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On Oct 06, 2011, at 12:33 PM, Carl Meyer wrote: I still think 'venv' is preferable to any of the other options proposed thus far. It's also nicely unique for googling. Funnily enough, the top hit right now for 'venv' is apparently Lua's project of the same name for the same purpose. Yeah, I meant to say that 'venv' also sounded like a reasonable choice to me (for the reasons Carl listed). Chers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] check for PyUnicode_READY look backwards
Amaury Forgeot d'Arc amauryfa at gmail.com writes: I'd prefer it was written : if (PyUnicode_READY(*filename) 0) because 0 clearly indicates an error condition. Why not just have it return 0 on error? This would be more consistent with API functions that return false values like NULL and would just be if (!PyUnicode_READY(s)) return NULL; in code. Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
On 06Oct2011 04:26, Glyph gl...@twistedmatrix.com wrote: | On Oct 5, 2011, at 10:46 PM, Cameron Simpson wrote: | Surely VERY FEW tests need to be run as root, and they need careful | consideration. The whole thing (build, full test suite) should | not run as root. | | This is news to me - is most of Python not supported to run as root? | I was under the impression that Python was supposed to run correctly as | root, and therefore there should be some buildbots dedicated to running | it that way. If only a few small parts of the API are supposed to work | perhaps this should be advertised more clearly in the documentation? Pretending the snark to be slightly serious: you've missed the point. The builtbots are building unreliable code, that being the point of the test suite. Doing unpredictable stuff as root is bad juju. Running the builtbots and their tests should not be run as root except for a very few special tests, and those few need careful consideration and sandboxing. | Ahem. Sorry for the snark, I couldn't resist. As terry more reasonably put it: | | running buildbot tests as root does not reflect the experience of | non-root users. It seems some tests need to be run both ways just for | correctness testing. | | (except I'd say all, not some) No. Terry is right and you are ... not. Most tests need no special privileges - they're testing language/library semantics that do not depend on the system facilities much, and when they do they should work for unprivileged users. Of course they _should_ work as root (barring the few tests like the issue cited, where things are expected to fail but don't because root is unconstrained by the permission system). HOWEVER, the whole suite should not be _tested_ as root because the code being testing is by definition untrusted. | Am I really the only person who feels unease about this scenario? | | More seriously: apparently you are not, but I am quite surprised by | that revelation. You should be :). The idea of root as a special, | magical place where real ultimate power resides is quite silly. root | is a title, like king. You're not just root, you're root _of_ | something. If the thing that you are root of is a dedicated virtual | machine with no interesting data besides the code under test, then this | is quite a lot like being a regular user in a similarly boring place. | It's like having the keys to an empty safe. Sadly, _no_. Root _is_ special, within the host and with scope to misbehave beyond the host. 1: The permission system does _not_ behave the same for root as for other users. 2: Root _can_ corrupt things anywhere in the system (within the VM, of course, but the builtbot is a subset of it). A normal unprivileged user will not have write permission to thing like: the OS image the compilers the system commands other user data areas all of which offer avenues to corrupt the built/test scenario. And if it is not a special purpose VM, the corrupt things for other uses and users of the system. 3: Root can also do other fun things like modify the network interfaces, including changing/adding IP addresss and MAC addresses. Which means that unless the VM (_if_ it is a VM) is running on a totally unroutable special purpose virtual network, it is possible to use the VM to pretend to be other machines on the same net and so forth. The prudent way to run the buildbots, especially if they cycle (refetch newer codebase, rebuilt, retest) instead of (scrub VM, reinstall, install built system, etc) is: - a user to fetch source and dispatch builds - possibly a distinct user to run the builds - definitely a distinct user to run the test suite And none of those be root. Cheers, -- Cameron Simpson c...@zip.com.au DoD#743 http://www.cskk.ezoshosting.com/cs/ Sorry, but at DoD minimum speed it is impossible to speak. There is just too much wind noise. At that speed I am spending all my concentration allowance on riding, and cannot afford anymore thought for words. However, when I finish a ride and the bike is in the garage cooling down, the single word that comes to mind is: BEER. - Jack Tavares, tavares@balrog, DoD#0570 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] check for PyUnicode_READY look backwards
On Thu, Oct 6, 2011 at 4:47 PM, Benjamin Peterson benja...@python.org wrote: Amaury Forgeot d'Arc amauryfa at gmail.com writes: I'd prefer it was written : if (PyUnicode_READY(*filename) 0) because 0 clearly indicates an error condition. Why not just have it return 0 on error? This would be more consistent with API functions that return false values like NULL and would just be if (!PyUnicode_READY(s)) return NULL; in code. Alas, that isn't the convention in C - courtesy of Unix, the convention is that for integer return codes, 0 means success. Yes, this is annoying, but violating it means you're not writing idiomatic C any more, you're trying to write Python-in-C. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
On Fri, 7 Oct 2011 08:27:01 +1100 Cameron Simpson c...@zip.com.au wrote: 2: Root _can_ corrupt things anywhere in the system (within the VM, of course, but the builtbot is a subset of it). A normal unprivileged user will not have write permission to thing like: the OS image the compilers the system commands other user data areas all of which offer avenues to corrupt the built/test scenario. And if it is not a special purpose VM, the corrupt things for other uses and users of the system. Why do you think it is not a special purpose VM? Also, if you think there's a security problem, why don't you take it in private with the buildbot owner instead of making such a fuss on a public mailing-list? The prudent way to run the buildbots, especially if they cycle (refetch newer codebase, rebuilt, retest) instead of (scrub VM, reinstall, install built system, etc) is: - a user to fetch source and dispatch builds - possibly a distinct user to run the builds - definitely a distinct user to run the test suite Your contribution is definitely welcome. Thanks Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] check for PyUnicode_READY look backwards
On Thu, 6 Oct 2011 17:40:20 -0400
Nick Coghlan ncogh...@gmail.com wrote:
On Thu, Oct 6, 2011 at 4:47 PM, Benjamin Peterson benja...@python.org wrote:
Amaury Forgeot d'Arc amauryfa at gmail.com writes:
I'd prefer it was written :
if (PyUnicode_READY(*filename) 0)
because 0 clearly indicates an error condition.
Why not just have it return 0 on error? This would be more consistent with
API
functions that return false values like NULL and would just be
if (!PyUnicode_READY(s)) return NULL;
in code.
Alas, that isn't the convention in C - courtesy of Unix, the
convention is that for integer return codes, 0 means success.
C is quite inconsistent, and so is our own C API.
if (PyUnicode_READY(s)) { ...}
definitely looks like the code block will be executed if the unicode
string is ready, though.
Regards
Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] check for PyUnicode_READY look backwards
2011/10/6 Benjamin Peterson benja...@python.org: Why not just have it return 0 on error? This would be more consistent with API functions that return false values like NULL and would just be if (!PyUnicode_READY(s)) return NULL; Most functions of the Python C API seems to follow one of two ways to indicate an error: - functions that return PyObject* will return NULL - functions that return an int will return -1 -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] check for PyUnicode_READY look backwards
Am 06.10.11 14:57, schrieb Amaury Forgeot d'Arc: Hi, with the new Unicode API, there are many checks like: +if (PyUnicode_READY(*filename)) +goto handle_error; I think you are misinterpreting what you are seeing. There are not *many* such checks. Of the PyUnicode_READY checks, 106 take the form if (PyUnicode_READY(foo) == -1) return NULL; 30 tests take the form that you mention. I believe all of those have been added by Victor, who just didn't follow the convention. So, Victor: please correct them. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Rename PyUnicode_KIND_SIZE ?
(also, is it useful? PEP 393 has added a flurry of new macros to unicodeobject.h and it's getting hard to know which ones are genuinely useful, or well-performing) Please understand that not all of them have been added by PEP 393 genuinely. Some have only be added by individual committers, and we have to review and revoke those that are not useful. (Of course, some of those that did get added by the PEP may also not be useful). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New stringbench benchmark results
Victor Stinner wrote:
Hum, copy-paste failure, I wrote numbers in the wrong order, it's:
(test: Python 3.2 = Python 3.3)
A.join([Bob]*100)): 0.92 = 2.11
(C+AB*300).rfind(CA): 0.57 = 1.03
(A + (Z*128*1024)).replace(A, BB, 1): 0.25 = 0.50
I improved str.replace(): it's now 5 times faster instead of 2 times slower
for this specific benchmark :-) (or 10 times faster in Python 3.3 before/after
my patch)
Talking about str.replace, I was surprised to see this behaviour in 3.2:
s = 'spam'
t = s.replace('a', 'a')
s is t
False
Given that strings are immutable, would it not be an obvious
optimization for replace to return the source string unchanged if the
old and new substrings are equal, and avoid making a potentially
expensive copy?
I note that if count is zero, the source string is returned unchanged:
t = s.replace('a', 'b', 0)
t is s
True
--
Steven
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
On Fri, Oct 07, 2011 at 08:27:01AM +1100, Cameron Simpson wrote: […] | running buildbot tests as root does not reflect the experience of | non-root users. It seems some tests need to be run both ways just for | correctness testing. | | (except I'd say all, not some) No. Terry is right and you are ... not. Most tests need no special privileges - they're testing language/library semantics that do not depend on the system facilities much, and when they do they should work for unprivileged users. You could also say that most tests that work on Linux work on FreeBSD too, so when they work on Linux they should work for FreeBSD too… so why bother running tests on FreeBSD at all? The reason is because the assumptions behind that “should” are wrong frequently enough to make it worth running tests in both environments. Like Glyph, I think that “running as root” is sufficiently different environment to “running as typical user” (in terms of how POSIX-like systems behave w.r.t. to things like permissions checks) to make it worthwhile to regularly run the whole test suite as root. HOWEVER, the whole suite should not be _tested_ as root because the code being testing is by definition untrusted. No, that just means you shouldn't trust *root*. Which is where a VM is a very useful tool. You can have the “as root” environment for your tests without the need to have anything important trust it. […] Root _is_ special, within the host and with scope to misbehave beyond the host. 1: The permission system does _not_ behave the same for root as for other users. Those are arguments *for* running tests as root! 2: Root _can_ corrupt things anywhere in the system (within the VM, of course, but the builtbot is a subset of it). A normal unprivileged user This appears to be a key error in your logic. There's no fundamental reason why “tests run as root inside a VM” must necessarily imply “buildbot process is run inside that same VM and is therefore vulnerable to code in that test run.” It may be more convenient to deploy it that way, but I'm sure it's possible to have a buildslave configured to e.g. start a pristine VM (from a snapshot with all the necessary build dependencies installed) and via SSH copy the the source into it, build it, run it, and report the results. The VM could be fully isolated from the real network and filesystem etc if you like. Given that it is certainly possible to run tests as root about as securely as running them without root, do you still feel it is not worth running the tests as root? The prudent way to run the buildbots, especially if they cycle (refetch newer codebase, rebuilt, retest) instead of (scrub VM, reinstall, install built system, etc) is: - a user to fetch source and dispatch builds - possibly a distinct user to run the builds - definitely a distinct user to run the test suite If we're talking prudence, then surely s/user/VM/ is even better :) -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
On 07Oct2011 12:46, Andrew Bennetts and...@bemusement.org wrote: | On Fri, Oct 07, 2011 at 08:27:01AM +1100, Cameron Simpson wrote: | […] | | running buildbot tests as root does not reflect the experience of | | non-root users. It seems some tests need to be run both ways just for | | correctness testing. | | | | (except I'd say all, not some) | | No. Terry is right and you are ... not. Most tests need no special | privileges - they're testing language/library semantics that do not | depend on the system facilities much, and when they do they should work | for unprivileged users. | | You could also say that most tests that work on Linux work on FreeBSD | too, so when they work on Linux they should work for FreeBSD too… so why | bother running tests on FreeBSD at all? The reason is because the | assumptions behind that “should” are wrong frequently enough to make it | worth running tests in both environments. For thoroughness, yes. | Like Glyph, I think that “running as root” is sufficiently different | environment to “running as typical user” (in terms of how POSIX-like | systems behave w.r.t. to things like permissions checks) to make it | worthwhile to regularly run the whole test suite as root. Hmm. Glyph seemed to be arguing both ways - that everything should be tested as root, and also that root is not special. I have unease over the former and disagreement over the latter. | HOWEVER, the whole suite should not be _tested_ as root because the code | being testing is by definition untrusted. | | No, that just means you shouldn't trust *root*. Which is where a VM is | a very useful tool. You can have the “as root” environment for your | tests without the need to have anything important trust it. | | Root _is_ special, within the host and with scope to misbehave beyond | the host. | 1: The permission system does _not_ behave the same for root as for | other users. | | Those are arguments *for* running tests as root! I think they're arguments for running _specific_ tests as root. File I/O based tests primarily I would suppose, though my first instinct would be to constrain even these to permission related tests. | 2: Root _can_ corrupt things anywhere in the system (within the VM, of | course, but the builtbot is a subset of it). A normal unprivileged user | | This appears to be a key error in your logic. There's no fundamental | reason why “tests run as root inside a VM” must necessarily imply | “buildbot process is run inside that same VM and is therefore vulnerable | to code in that test run.” Indeed, I had no considered that the tests might be run in a special VM distinct from the build environment. In that setup most of my concerns are moot. | It may be more convenient to deploy it that way, but I'm sure it's | possible to have a buildslave configured to e.g. start a pristine VM | (from a snapshot with all the necessary build dependencies installed) and | via SSH copy the the source into it, build it, run it, and report the | results. The VM could be fully isolated from the real network and | filesystem etc if you like. | | Given that it is certainly possible to run tests as root about as | securely as running them without root, do you still feel it is not worth | running the tests as root? Not in the style you describe above. To clarify: I agree with your suggestion. | The prudent way to run the buildbots, especially if they cycle (refetch | newer codebase, rebuilt, retest) instead of (scrub VM, reinstall, | install built system, etc) is: | |- a user to fetch source and dispatch builds |- possibly a distinct user to run the builds |- definitely a distinct user to run the test suite | | If we're talking prudence, then surely s/user/VM/ is even better :) Yes. And a distinct VM instance for the root and non-root tests, too. Cheers, -- Cameron Simpson c...@zip.com.au DoD#743 http://www.cskk.ezoshosting.com/cs/ all coders are created equal; that they are endowed with certain unalienable rights, of these are beer, net connectivity, and the pursuit of bugfixes... - Gregory R Block ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
Cameron Simpson wrote: On 06Oct2011 04:26, Glyph gl...@twistedmatrix.com wrote: | On Oct 5, 2011, at 10:46 PM, Cameron Simpson wrote: | Surely VERY FEW tests need to be run as root, and they need careful | consideration. The whole thing (build, full test suite) should | not run as root. | | This is news to me - is most of Python not supported to run as root? | I was under the impression that Python was supposed to run correctly as | root, and therefore there should be some buildbots dedicated to running | it that way. If only a few small parts of the API are supposed to work | perhaps this should be advertised more clearly in the documentation? Pretending the snark to be slightly serious: you've missed the point. The builtbots are building unreliable code, that being the point of the test suite. Doing unpredictable stuff as root is bad juju. Sorry Cameron, it seems to me that you have missed the point, not Glyph. If the builtbots were predictable, there would be no point in running them because we would already know the answer. But since they *might* fail, they need to be run: that is an argument in favour of tests run explicitly as root, not against it. Since Python running as root is supported[1], then it must be tested running as root. What's the alternative? Wait for some user to report that Python nuked their production system when run as root? Better that we find out first hand. It would be embarrassing enough if (say) list.append crashed our test system. How much worse would if be if the first time we found out about it was after it did so to Google's production servers? To put it another way: Doing unpredictable stuff as root on a production machine is bad juju. Doing unpredictable stuff as root in order to find out what it will do *before* putting it into production is absolutely vital. Running the builtbots and their tests should not be run as root except for a very few special tests, and those few need careful consideration and sandboxing. Are you suggested that they aren't currently sandboxed? | Ahem. Sorry for the snark, I couldn't resist. As terry more reasonably put it: | | running buildbot tests as root does not reflect the experience of | non-root users. It seems some tests need to be run both ways just for | correctness testing. | | (except I'd say all, not some) No. Terry is right and you are ... not. Most tests need no special privileges - they're testing language/library semantics that do not depend on the system facilities much, and when they do they should work for unprivileged users. Of course they _should_ work as root (barring the few tests like the issue cited, where things are expected to fail but don't because root is unconstrained by the permission system). HOWEVER, the whole suite should not be _tested_ as root because the code being testing is by definition untrusted. So what you are saying is that the most critical situation, with the greatest consequences if there is a failure, should *not* be tested, but taken on trust that it will just work as expected. This makes no sense to me. I would say that testing as root is more important, not less, because the consequences of unexpected failure is so much worse. It seems to me that you are putting the security of the build-bot ahead of people's real-life production systems, that you expect them to run *untested code* in production as root. To me, this seems wrong. What exactly is your fear? That Python run as root will be able to escape the jail it is running in and do bad things to the host machine? That's a legitimate concern, but that's an argument to be taken up with the sys admins who set up the jail. It's not an argument against testing as root. It's an argument against testing as root *badly*. [1] If not, that will come as a mighty big surprise to Red Hat, among others, who use Python for system tools run as root. -- Steven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
On 07Oct2011 13:42, Steven D'Aprano st...@pearwood.info wrote: | Cameron Simpson wrote: | On 06Oct2011 04:26, Glyph gl...@twistedmatrix.com wrote: | | On Oct 5, 2011, at 10:46 PM, Cameron Simpson wrote: | | Surely VERY FEW tests need to be run as root, and they need careful | | consideration. The whole thing (build, full test suite) should | | not run as root. | | | This is news to me - is most of Python not supported to run as | root? | | I was under the impression that Python was supposed to run correctly as | | root, and therefore there should be some buildbots dedicated to running | | it that way. If only a few small parts of the API are supposed to work | | perhaps this should be advertised more clearly in the documentation? | | Pretending the snark to be slightly serious: you've missed the point. | The builtbots are building unreliable code, that being the point of the | test suite. Doing unpredictable stuff as root is bad juju. | | Sorry Cameron, it seems to me that you have missed the point, not | Glyph. We're probably both aiming badly. See my reply to Andrew Bennetts; I'm less concerned if his described scenario is typical. [...snip...] | Doing unpredictable stuff as root on a production machine is bad | juju. Doing unpredictable stuff as root in order to find out what it | will do *before* putting it into production is absolutely vital. Yes yes yes. | Running the builtbots and their tests should not be run as root except | for a very few special tests, and those few need careful consideration | and sandboxing. | | Are you suggested that they aren't currently sandboxed? No, but it was my instinctive fear. Please see my reply to Andrew Bennetts. I find nothing to disagree with in your reply. Cheers, -- Cameron Simpson c...@zip.com.au DoD#743 http://www.cskk.ezoshosting.com/cs/ The word is not the thing. The map is not the territory. The symbol is not the thing symbolized. - S.I. Hayakawa ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] check for PyUnicode_READY look backwards
Benjamin Peterson wrote: Why not just have it return 0 on error? This would be more consistent with API functions that return false values like NULL But that would make it confusingly different from all the other functions that return ints. The NULL convention is only used when the function returns a pointer. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New stringbench benchmark results
Steven D'Aprano wrote: Given that strings are immutable, would it not be an obvious optimization for replace to return the source string unchanged if the old and new substrings are equal, Only if this situation occurs frequently enough to outweigh the overhead of comparing the target and replacement strings. This check could be performed very cheaply when both strings are interned, so it might be worth doing in that case. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
On Oct 6, 2011, at 10:11 PM, Cameron Simpson wrote: Hmm. Glyph seemed to be arguing both ways - that everything should be tested as root, and also that root is not special. I have unease over the former and disagreement over the latter. Your reply to Stephen suggests that we are actually in agreement, but just to be clear: I completely understand that root is special in that the environment allows for several behaviors which are not true for a normal user. Which is precisely why it must be tested by a (properly sandboxed) buildbot :). It's just not special in the sense that having root on a throwaway VM would allow you to do non-throwaway things. The one thing one must always be careful of, of course, is having your bandwidth chewed up for some nefarious purpose (spam, phishing) but that sort of thing should be caught with other monitoring tools. Plus, there are lots of other impediments to getting Python's buildbots to do something nasty. Only people with a commit bit should be able to actually push changes that buildbot will see. So avoiding root is more about avoiding mistakes than avoiding attacks. (After all, if this process isn't completely secure, then neither is the Python that's shipped in various OSes: in which case, game over _everywhere_.) Finally, and unfortunately, there are so many privilege escalation exploits in so many different daemons and applications that it's foolish to treat root as too terribly special: unless you're a real hardening expert and you spend a lot of effort keeping up to the second on security patches, the ability to execute completely arbitrary untrusted code as a unprivileged local user on your system can likely be converted with little effort into the ability to execute arbitrary untrusted code as root. Although, ironically, buildbots are often minimally configured and don't run any other services, so maybe these environments are one of the few places where it actually does make a difference :-). (Which is precisely why all daemons everywhere should be written in Python. Buffer overflows are dumb, it's 2011 already, come on. Use Twisted.)___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
