Re: [Python-Dev] [Python-checkins] r88904 - tracker/instances/python-dev/html/issue.item.js
Hi, On 07/10/2011 10.02, ezio.melotti wrote: Author: ezio.melotti Date: Fri Oct 7 09:02:07 2011 New Revision: 88904 Log: #422: add keyword shortcuts to navigate through the messages and to reply. I added keyboard shortcut to navigate through the messages in the tracker (yes, keyboard, not keyword, that's a typo :). There are two groups of shortcuts available: * mnemonics: f: first message; p: previous message; n: next message; l: last message; r: reply (jumps on the comment field and focuses it); * vim-style: h: left (first message); k: up (previous message); j: down (next message); l: right (last message); i: insert-mode (jumps on the comment field and focuses it); esc: normal-mode (unfocus the field and re-enables the commands); The two groups don't conflict with each other, so all the keys always work. The shortcuts don't require key combinations like ctrl+f/alt+f -- 'f' is enough. The shortcuts are available only in the issue page, and not in the main page with the list of issues. The shortcuts use javascript, so they won't work if js is disabled. The issue is tracked here: http://psf.upfronthosting.co.za/roundup/meta/issue422 If you have any problem/feedback reply either there or here. A few notes about the change: * The 'end' key doesn't jump to the last message anymore. The normal browser behavior (i.e. go to the end of the page) is now restored. Use 'l' (last) to jump to the last message. * The patch conflicts with the browser 'find-as-you-type' if the first letter is a shortcut. If you are using the find-as-you-type, use ctrl+f instead. * f/l *always* jump to the first/last message, regardless of the position of the page. p/n use an index that does *not* change when you scroll, and do nothing on the first/last message respectively. If you are at the second-last message, scroll to the top, and hit 'n', you will still jump to the last message. If you are at the last message, scroll to the top, and hit 'n' the page won't scroll (you can use 'l' instead). * I added the shortcuts to the left sidebar but I plan to move them to the devguide eventually. * It might be useful to add a shortcut to submit. 's' would be a good candidate, but it might be hit accidentally (an are you sure? popup might solve this). ctrl+enter/ctrl+s might be better, but they might conflict with the browser commands. * While replying (i.e. while writing in the comment textarea), the shortcuts are disabled. You can hit ESC to unfocus the textarea and then use them. You can then press 'r' again to continue editing. Best Regards, Ezio Melotti ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] check for PyUnicode_READY look backwards
Le 07/10/2011 00:20, Martin v. Löwis a écrit : Am 06.10.11 14:57, schrieb Amaury Forgeot d'Arc: Hi, with the new Unicode API, there are many checks like: + if (PyUnicode_READY(*filename)) + goto handle_error; I think you are misinterpreting what you are seeing. There are not *many* such checks. Of the PyUnicode_READY checks, 106 take the form if (PyUnicode_READY(foo) == -1) return NULL; 30 tests take the form that you mention. I believe all of those have been added by Victor, who just didn't follow the convention. Yes, I wrote if (PyUnicode_READY(foo)), but I agree that it is confusing when you read the code, especially because we have also a PyUnicode_IS_READY(foo) macro! if (!PyUnicode_READY(foo)) is not better, also because of PyUnicode_IS_READY(foo). I prefer PyUnicode_IS_READY(foo) 0 over PyUnicode_IS_READY(foo) == -1. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] check for PyUnicode_READY look backwards
Victor Stinner victor.stin...@haypocalc.com wrote: Yes, I wrote if (PyUnicode_READY(foo)), but I agree that it is confusing when you read the code, especially because we have also a PyUnicode_IS_READY(foo) macro! if (!PyUnicode_READY(foo)) is not better, also because of PyUnicode_IS_READY(foo). I prefer PyUnicode_IS_READY(foo) 0 over PyUnicode_IS_READY(foo) == -1. Do you mean PyUnicode_READY(foo) 0? I also prefer that idiom. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] check for PyUnicode_READY look backwards
Le 07/10/2011 10:07, Stefan Krah a écrit : Victor Stinnervictor.stin...@haypocalc.com wrote: Yes, I wrote if (PyUnicode_READY(foo)), but I agree that it is confusing when you read the code, especially because we have also a PyUnicode_IS_READY(foo) macro! if (!PyUnicode_READY(foo)) is not better, also because of PyUnicode_IS_READY(foo). I prefer PyUnicode_IS_READY(foo) 0 over PyUnicode_IS_READY(foo) == -1. Do you mean PyUnicode_READY(foo) 0? I also prefer that idiom. Oops, yes I mean PyUnicode_READY(foo) 0. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] counterintuitive behavior (bug?) in Counter with +=
2011/10/6 Petri Lehtinen pe...@digip.org: Lars Buitinck wrote: from collections import Counter a = Counter([1,2,3]) b = a a += Counter([3,4,5]) a is b False Sounds like a good idea to me. You should open an issue in the tracker at http://bugs.python.org/. Done that: http://bugs.python.org/issue13121 -- Lars Buitinck Scientific programmer, ILPS University of Amsterdam ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
Andrew Bennetts writes: No, that just means you shouldn't trust *root*. Which is where a VM is a very useful tool. You can have the “as root” environment for your tests without the need to have anything important trust it. Cameron acknowledges that he missed that. So maybe he was right for the wrong reason; he's still right. But in the current context, it is not an argument for not worrying, because there is no evidence at all that the OP set up his buildbot in a secure sandbox. As I read his followups, he simply didn't bother to set up an unprivileged user and run the 'bot as that user. 2: Root _can_ corrupt things anywhere in the system (within the VM, of course, but the builtbot is a subset of it). A normal unprivileged user This appears to be a key error in your logic. There's no fundamental reason why “tests run as root inside a VM” must necessarily imply “buildbot process is run inside that same VM and is therefore vulnerable to code in that test run.” Cameron's logic is correct. When security is in question, one must assume that *everything* is vulnerable until proven otherwise. Is secure requires a universal quantifier; is insecure only an existential one. The principle here is ran as root without further explanation is a litmus test for not bothering about security, even today. It's worth asking for explanation, or at least a comment that all the buildbot contributors I've talked to have put a lot of effort into security configuration. It may be more convenient to deploy it that way, You bet, and hundreds of thousands of viruses exploiting IE thank Microsoft for its devotion to convenience. While much can be done to make secure configuration more convenient than it is, nevertheless the state of the art is that convenience is the enemy of security. but I'm sure it's possible to have a buildslave configured to e.g. start a pristine VM (from a snapshot with all the necessary build dependencies installed) and via SSH copy the the source into it, build it, run it, and report the results. The VM could be fully isolated from the real network and filesystem etc if you like. Sure it's possible. In security, the question is never can it be done?; it's was it done? We have *no* evidence justifying an *assumption* that it was done in the current case, or for other buildbots, for that matter. In fact, as I read the OP's followups, that was *not* the case; the assumption is falsified. Nevertheless, several people who I would have thought would know better are *all* arguing from the assumption that the OP configured his test system with security (rather than convenience) in mind, and are castigating Cameron for *not* making that same assumption. To my mind, every post is increasing justification for his unease. :-( And that's why this thread belongs on this list, rather than on Bruce Schneier's blog. It's very easy these days to set up a basic personal VM, and folk of goodwill will do so to help the project with buildbots to provide platform coverage in testing new code. But this contribution involves certain risks (however low probability, some Very Bad Things *could* happen). Contributors should get help in evaluating the potential threats and corresponding risks, and in proper configuration. Not assurances that nothing will go wrong because you probably run the 'bot in a VM. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
On Oct 7, 2011, at 5:10 AM, Stephen J. Turnbull wrote: The principle here is ran as root without further explanation is a litmus test for not bothering about security, even today. It's worth asking for explanation, or at least a comment that all the buildbot contributors I've talked to have put a lot of effort into security configuration. This is a valid point. I think that Cameron and I may have had significantly different assumptions about the environment being discussed here. I may have brought some assumptions about the build farm here that don't actually apply to the way Python does it. To sum up what I believe is now the consensus from this thread: Anyone setting up a buildslave should take care to invoke the build in an environment where an out-of-control buildbot, potentially executing arbitrarily horrible and/or malicious code, should not damage anything. Builders should always be isolated from valuable resources, although the specific mechanism of isolation may differ. A virtual machine is a good default, but may not be sufficient; other tools for cutting of the builder from the outside world would be chroot jails, solaris zones, etc. Code runs differently as privileged vs. unprivileged users. Therefore builders should be set up in both configurations, running the full test suite, to ensure that all code runs as expected in both configurations. Some tests, as the start of this thread indicates, must have some special logic to make sure they do or do not run, or run differently, in privileged vs. unprivileged configurations, but generally speaking most things should work in both places. Access to root my provide access to slightly surprising resources, even within a VM (such as the ability to send spoofed IP packets, change the MAC address of even virtual ethernet cards, etc), and administrators should be aware that this is the case when configuring the host environment for a run-as-root builder. You don't want to end up with a compromised test VM that can snoop on your network. Have I left anything out? :-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
On 07Oct2011 06:18, Glyph gl...@twistedmatrix.com wrote: | On Oct 7, 2011, at 5:10 AM, Stephen J. Turnbull wrote: | | The principle here is ran as root without further explanation is a | litmus test for not bothering about security, even today. It's | worth asking for explanation, or at least a comment that all the | buildbot contributors I've talked to have put a lot of effort into | security configuration. | | This is a valid point. I think that Cameron and I may have | had significantly different assumptions about the environment being | discussed here. I may have brought some assumptions about the build | farm here that don't actually apply to the way Python does it. Likewise. I state now that I have no actual knowledge of the practices in the build farm(s). | To sum up what I believe is now the consensus from this thread: | | Anyone setting up a buildslave should take care to invoke the build in | an environment where an out-of-control buildbot, potentially executing | arbitrarily horrible and/or malicious code, should not damage anything. | Builders should always be isolated from valuable resources, although | the specific mechanism of isolation may differ. A virtual machine is a | good default, but may not be sufficient; other tools for cutting of the | builder from the outside world would be chroot jails, solaris zones, etc. | | Code runs differently as privileged vs. unprivileged users. Therefore | builders should be set up in both configurations, running the full test | suite, to ensure that all code runs as expected in both configurations. | Some tests, as the start of this thread indicates, must have some | special logic to make sure they do or do not run, or run differently, | in privileged vs. unprivileged configurations, but generally speaking | most things should work in both places. | | Access to root my provide access to slightly surprising resources, | even within a VM (such as the ability to send spoofed IP packets, | change the MAC address of even virtual ethernet cards, etc), and | administrators should be aware that this is the case when configuring | the host environment for a run-as-root builder. You don't want to end | up with a compromised test VM that can snoop on your network. | | Have I left anything out? :-) I think that the build and the tests should be different security scopes/zones/levels: different users or different VMs. Andrew's suggestion of a VM-for-tests sounds especially good. And that I think the as-root tests suite shouldn't run unless the not-root test suite passes. Cheers, -- Cameron Simpson c...@zip.com.au DoD#743 http://www.cskk.ezoshosting.com/cs/ It is not true that life is one damn thing after another -- it's one damn thing over and over.- Edna St. Vincent Millay ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
On Oct 7, 2011, at 6:40 AM, Cameron Simpson wrote: I think that the build and the tests should be different security scopes/zones/levels: different users or different VMs. Andrew's suggestion of a VM-for-tests sounds especially good. To me, build and test are largely the same function, since a build whose tests haven't been run is just a bag of bits :). But in the sense that root should never be required to do a build, I don't see a reason to bother supporting that configuration: it makes sense to always do the build as a regular user. And that I think the as-root tests suite shouldn't run unless the not-root test suite passes. Why's that? The as-root VM needs to be equally secure either way, and it's a useful data point to see that the as-root tests *didn't* break, if they didn't; this way a developer can tell at a glance that the failure is either a test that needs to be marked as 'root only' or a change that causes permissions to be required that it shouldn't have. (In general I object to suggestions of the form don't run the tests unless X, unless X is a totally necessary pre-requisite like the compile finished.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
On 07Oct2011 06:50, Glyph gl...@twistedmatrix.com wrote:
| On Oct 7, 2011, at 6:40 AM, Cameron Simpson wrote:
| I think that the build and the tests should be different security
| scopes/zones/levels: different users or different VMs. Andrew's
| suggestion of a VM-for-tests sounds especially good.
|
| To me, build and test are largely the same function, since a build
| whose tests haven't been run is just a bag of bits :). But in the sense
| that root should never be required to do a build, I don't see a reason
| to bother supporting that configuration: it makes sense to always do
| the build as a regular user.
I don't mean buid as root and test as regular user, I mean build as regular
user and test as different user. This can be used to prevent the test user
from having write permission to the built code.
My thinking is that the build is a well defined set of safe
operations: copy the source (safe, just a data copy), compile the source
(safe, presuming bug-free compiler). Of course I'm glossing over any
autoconfiguration shell scripts and makefiles full of source-code-supplied
shell commands - nasty nasty.
Basicly I was taking the view that a build should be a safe source
code to machine code translation process.
By contrast, the tests _run_ the somewhat-unknown test suite. Not an
inherently safe procedure.
Think of the build being like a PDF viewer rendering a document to the
display. And the tests as being the user reading a list of instructions
off that display and doing stuff. It ought to be safe to render the PDF;
the user's actions are unsafe.
| And that I think the as-root tests suite shouldn't run unless the
| not-root test suite passes.
|
| Why's that? The as-root VM needs to be equally secure either way,
| and it's a useful data point to see that the as-root tests *didn't*
| break, if they didn't; this way a developer can tell at a glance that
| the failure is either a test that needs to be marked as 'root only'
| or a change that causes permissions to be required that it shouldn't have.
|
| (In general I object to suggestions of the form don't run the tests
| unless X, unless X is a totally necessary pre-requisite like the
| compile finished.)
Suppose a test is dangerously broken through ineptitude or even malice.
Extreme example: a test makes a bunch of test files and cleans up with
rm -r /. (Non-malicious scenario: rm -r ${testdatatree}/, with
$testdatatree accidentally undefined, eg through a typo.)
Such a test will fail when unprivileged. (Of course a mlicious test
might say do not set off the bomb unless I am root:-)
The point here is security, not test coverage: if a procedure is known
to be broken as a regular user, is it not highly unsafe to then run it
as root?
Cheers,
--
Cameron Simpson c...@zip.com.au DoD#743
http://www.cskk.ezoshosting.com/cs/
1st Law Economists: For every economist there exists an equal and opposite
economist.
2nd Law Economists: They're both always wrong!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] socket module build failure
I work on Ubuntu Jaunty for my cpython development work - an old version, I know, but still quite serviceable and has worked well for me over many months. With the latest default cpython repository, however, I can't run the regression suite because the socket module now fails to build: gcc -pthread -fPIC -g -O0 -Wall -Wstrict-prototypes -IInclude -I. -I./Include -I/usr/local/include -I/home/vinay/projects/python/default -c /home/vinay/projects/python/default/Modules/socketmodule.c -o build/temp.linux-i686-3.3-pydebug/home/vinay/projects/python/default /Modules/socketmodule.o .../Modules/socketmodule.c: In function ‘makesockaddr’: .../Modules/socketmodule.c:1224: error: ‘AF_CAN’ undeclared (first use in this function) .../Modules/socketmodule.c:1224: error: (Each undeclared identifier is reported only once .../Modules/socketmodule.c:1224: error: for each function it appears in.) .../Modules/socketmodule.c: In function ‘getsockaddrarg’: .../Modules/socketmodule.c:1610: error: ‘AF_CAN’ undeclared (first use in this function) .../Modules/socketmodule.c: In function ‘getsockaddrlen’: .../Modules/socketmodule.c:1750: error: ‘AF_CAN’ undeclared (first use in this function) On this system, AF_CAN *is* defined, but in linux/socket.h, not in sys/socket.h. From what I can see, sys/socket.h includes bits/socket.h which includes asm/socket.h, but apparently linux/socket.h isn't included. Is this a bug which doesn't show up on more recent Linux versions, or is Jaunty no longer supported for Python development, or could something be wrong with my configuration? BTW nothing has changed on the machine other than updates to Jenkins and the cpython repo. Any advice would be appreciated! Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
On Oct 7, 2011, at 7:10 AM, Cameron Simpson wrote: The point here is security, not test coverage: if a procedure is known to be broken as a regular user, is it not highly unsafe to then run it as root? No. As I mentioned previously, any environment where the tests are run should be isolated from any resources that are even safety-relevant, let alone safety-critical, whether they're running as a regular user _or_ root. In theory, one might automatically restore the run-as-root buildslave VM from a snapshot before every single test run. In practice this is probably too elaborate to bother with and an admin can just hit the 'restore' button in the fairly unlikely case that something does happen to break the buildslave. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] socket module build failure
Is this a bug which doesn't show up on more recent Linux versions Probably. AF_CAN was introduced in e767318baccd. Cheers Ross ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] socket module build failure
Hello, 2011/10/7 Vinay Sajip vinay_sa...@yahoo.co.uk: I work on Ubuntu Jaunty for my cpython development work - an old version, I know, but still quite serviceable and has worked well for me over many months. With the latest default cpython repository, however, I can't run the regression suite because the socket module now fails to build: It's due to the recent inclusion of PF_CAN support: http://hg.python.org/cpython/rev/e767318baccd It looks like your header files are different from what's found in other distributions. Please reopen issue #10141, we'll try to go from there. Cheers, cf ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] counterintuitive behavior (bug?) in Counter with +=
On Oct 3, 2011, at 6:12 AM, Lars Buitinck wrote: After some digging, I found out that Counter [2] does not have __iadd__ and += copies the entire left-hand side in __add__! This seems like a reasonable change for Py3.3. I also figured out that I should use the update method instead, which I will, but I still find that uglier than +=. I would submit a patch to implement __iadd__, but I first want to know if that's considered the right behavior, since it changes the semantics of +=: Yes, update() is the fastest way. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New stringbench benchmark results
Le 07/10/2011 03:19, Steven D'Aprano a écrit : Given that strings are immutable, would it not be an obvious optimization for replace to return the source string unchanged if the old and new substrings are equal, and avoid making a potentially expensive copy? I just implemented this optimization in 9c1b76936b79, but only if old and new substrings are the same object (old is new). *Compare* substrings (content) would slow down .replace() in most cases. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] check for PyUnicode_READY look backwards
if (!PyUnicode_READY(foo)) is not better, also because of PyUnicode_IS_READY(foo). I prefer PyUnicode_IS_READY(foo) 0 over PyUnicode_IS_READY(foo) == -1. Ok, so feel free to replace all == -1 tests with 0 tests as well. I'll point out that the test for -1 is also widespread in Python, e.g. when checking return values from PyObject_SetAttrString, BaseException_init, PyThread_create_key, PyObject_DelAttrString, etc. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] SimpleHTTPServer slashdot (Was: Python Core Tools)
On Sun, Oct 2, 2011 at 3:17 PM, Maciej Fijalkowski fij...@gmail.com wrote: On Sun, Oct 2, 2011 at 8:05 AM, Maciej Fijalkowski fij...@gmail.com wrote: On Sun, Oct 2, 2011 at 5:02 AM, anatoly techtonik techto...@gmail.com wrote: Hello, I've stumbled upon Dave Beazley's article [1] about trying ancient GIL removal patch at http://dabeaz.blogspot.com/2011/08/inside-look-at-gil-removal-patch-of.html and looking at the output of Python dis module thought that it would be cool if there were tools to inspect, explain and play with Python bytecode. Little visual assembler, that shows bytecode and disassembly side by side and annotates the listing with useful hints (like interpreter code optimization decisions). That will greatly help many new people understand how Python works and explain complicated stuff like GIL and stackless by copy/pasting pictures from there. PyPy has a tool named 'jitviewer' [2] that may be that I am looking for, but the demo is offline. I put demo back online. It's just that SimpleHTTPServer doesn't quite survive slashdot effect. Where do I fill a bug report :) http://bugs.python.org Is the demo address is still http://wyvern.cs.uni-duesseldorf.de:5000/ ? Still can't connect. =| -- anatoly t. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SimpleHTTPServer slashdot (Was: Python Core Tools)
On Fri, Oct 7, 2011 at 08:57, anatoly techtonik techto...@gmail.com wrote: It's just that SimpleHTTPServer doesn't quite survive slashdot effect. Where do I fill a bug report :) http://bugs.python.org http://www.theonion.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] check for PyUnicode_READY look backwards
On Fri, Oct 7, 2011 at 9:21 AM, Martin v. Löwis mar...@v.loewis.de wrote: if (!PyUnicode_READY(foo)) is not better, also because of PyUnicode_IS_READY(foo). I prefer PyUnicode_IS_READY(foo) 0 over PyUnicode_IS_READY(foo) == -1. Ok, so feel free to replace all == -1 tests with 0 tests as well. I'll point out that the test for -1 is also widespread in Python, e.g. when checking return values from PyObject_SetAttrString, BaseException_init, PyThread_create_key, PyObject_DelAttrString, etc. FWIW, I don't mind whether it's 0 or == -1, so long as there's a comparison there to kick my brain out of Python boolean logic mode. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SimpleHTTPServer slashdot (Was: Python Core Tools)
On Fri, 7 Oct 2011 09:04:08 -0500 Brian Curtin brian.cur...@gmail.com wrote: On Fri, Oct 7, 2011 at 08:57, anatoly techtonik techto...@gmail.com wrote: It's just that SimpleHTTPServer doesn't quite survive slashdot effect. Where do I fill a bug report :) http://bugs.python.org http://www.theonion.com/ Does theonion.com really run a SimpleHTTPServer, or am I missing something? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
Glyph writes: Have I left anything out? :-) Probably. That's the nature of the problem. But you caught enough that if all our buildbots are set up that way, the Bad Guys' scripts will probably conclude there's nothing to see here, and move along. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New stringbench benchmark results
Le 06/10/2011 12:42, Victor Stinner a écrit : A.join([Bob]*100)): 0.92 = 2.11 I just optimized PyUnicode_Join() for such dummy benchmark. It's now 1.2x slower instead of 2.3x slower on this dummy benchmark. With longer *ASCII* strings, Python 3.3 is now 2x (narrow 3.2) or 4x (wide 3.2) faster than Python 3.2. For example with this micro-benchmark: ./python -m timeit 'x=[x*500]*5000; y=\n; z=y.join' 'z(x)' Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2011-09-30 - 2011-10-07)
Python tracker at http://bugs.python.org/
To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.
Issues counts and deltas:
open3052 ( +6)
closed 21853 (+40)
total 24905 (+46)
Open issues with patches: 1302
Issues opened (33)
==
#4147: xml.dom.minidom toprettyxml: omit whitespace for text-only ele
http://bugs.python.org/issue4147 reopened by ezio.melotti
#10141: SocketCan support
http://bugs.python.org/issue10141 reopened by neologix
#11250: 2to3 truncates files at formfeed character
http://bugs.python.org/issue11250 reopened by barry
#12210: test_smtplib: intermittent failures on FreeBSD
http://bugs.python.org/issue12210 reopened by skrah
#12804: make test fails on systems without internet access
http://bugs.python.org/issue12804 reopened by pitrou
#13078: Python Crashes When Saving Or Opening
http://bugs.python.org/issue13078 opened by Ash.Sparks
#13081: Crash in Windows with unknown cause
http://bugs.python.org/issue13081 opened by rlibiez
#13083: _sre: getstring() releases the buffer before using it
http://bugs.python.org/issue13083 opened by haypo
#13086: Update howto/cporting.rst so it talks about 3.x instead of 3.0
http://bugs.python.org/issue13086 opened by larry
#13088: Add Py_hexdigits constant: use one unique constant to format a
http://bugs.python.org/issue13088 opened by haypo
#13089: parsetok.c: memory leak
http://bugs.python.org/issue13089 opened by skrah
#13090: posix_read: memory leak
http://bugs.python.org/issue13090 opened by skrah
#13091: ctypes: memory leak
http://bugs.python.org/issue13091 opened by skrah
#13092: pep-393: memory leaks #2
http://bugs.python.org/issue13092 opened by skrah
#13093: Redundant code in PyUnicode_EncodeDecimal()
http://bugs.python.org/issue13093 opened by skrah
#13094: Need Programming FAQ entry for the behavior of closures
http://bugs.python.org/issue13094 opened by Tomáš.DvoÅák
#13096: ctypes: segfault with large POINTER type names
http://bugs.python.org/issue13096 opened by meador.inge
#13097: ctypes: segfault with large number of callback arguments
http://bugs.python.org/issue13097 opened by meador.inge
#13100: sre_compile._optimize_unicode() needs a cleanup
http://bugs.python.org/issue13100 opened by haypo
#13101: Module Doc viewer closes when browser window closes on Windows
http://bugs.python.org/issue13101 opened by brian.curtin
#13102: xml.dom.minidom does not support default namespaces
http://bugs.python.org/issue13102 opened by crass
#13103: copy of an asyncore dispatcher causes infinite recursion
http://bugs.python.org/issue13103 opened by xdegaye
#13105: Please elaborate on how 2.x and 3.x are different heads
http://bugs.python.org/issue13105 opened by larry
#13107: Text width in optparse.py can become negative
http://bugs.python.org/issue13107 opened by adambyrtek
#13111: Error 2203 when installing Python/Perl?
http://bugs.python.org/issue13111 opened by MA.S
#13114: check -r fails with non-ASCII unicode long_description
http://bugs.python.org/issue13114 opened by Cykooz
#13115: tp_as_{number,sequence,mapping} can't be set using PyType_From
http://bugs.python.org/issue13115 opened by awilkins
#13116: setup.cfg in [sb]dists should be static
http://bugs.python.org/issue13116 opened by eric.araujo
#13119: Newline for print() is \n on Windows, and not \r\n as expected
http://bugs.python.org/issue13119 opened by M..Z.
#13120: Default nosigint option to pdb.Pdb() prevents use in non-main
http://bugs.python.org/issue13120 opened by bpb
#13121: collections.Counter's += copies the entire object
http://bugs.python.org/issue13121 opened by larsmans
#13122: Out of date links in the sidebar of the documentation index of
http://bugs.python.org/issue13122 opened by smarnach
#13123: bdist_wininst uninstaller does not remove pycache directories
http://bugs.python.org/issue13123 opened by pmoore
Most recent 15 issues with no replies (15)
==
#13123: bdist_wininst uninstaller does not remove pycache directories
http://bugs.python.org/issue13123
#13122: Out of date links in the sidebar of the documentation index of
http://bugs.python.org/issue13122
#13120: Default nosigint option to pdb.Pdb() prevents use in non-main
http://bugs.python.org/issue13120
#13116: setup.cfg in [sb]dists should be static
http://bugs.python.org/issue13116
#13115: tp_as_{number,sequence,mapping} can't be set using PyType_From
http://bugs.python.org/issue13115
#13111: Error 2203 when installing Python/Perl?
http://bugs.python.org/issue13111
#13107: Text width in optparse.py can become negative
http://bugs.python.org/issue13107
#13100: sre_compile._optimize_unicode() needs a cleanup
http://bugs.python.org/issue13100
#13097: ctypes: segfault with large number of callback arguments
http://bugs.python.org/issue13097
#13093: Redundant code in
Re: [Python-Dev] Status of the built-in virtualenv functionality in 3.3
Hi, I too prefer venv (module) and pyvenv (script). Regards ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (2.7): Issue #7367: Add test case to test_pkgutil for walking path with
Hi Ned, Issue #7367: Add test case to test_pkgutil for walking path with an unreadable directory. Kudos for fixing this bug, the pydoc one and cleaning the duplicate reports! diff --git a/Lib/test/test_pkgutil.py b/Lib/test/test_pkgutil.py --- a/Lib/test/test_pkgutil.py +++ b/Lib/test/test_pkgutil.py @@ -78,6 +78,17 @@ +def test_unreadable_dir_on_syspath(self): +# issue7367 - walk_packages failed if unreadable dir on sys.path +package_name = unreadable_package +d = os.path.join(self.dirname, package_name) +# this does not appear to create an unreadable dir on Windows +# but the test should not fail anyway +os.mkdir(d, 0) +for t in pkgutil.walk_packages(path=[self.dirname]): +self.fail(unexpected package found) +os.rmdir(d) This should use a try/finally block (or self.addCleanup, my preference) to make sure rmdir is always called. Regards ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] More Buildbot Information in Devguide (Was: Re: cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as)
On Fri, Oct 7, 2011 at 4:18 AM, Glyph gl...@twistedmatrix.com wrote: On Oct 7, 2011, at 5:10 AM, Stephen J. Turnbull wrote: The principle here is ran as root without further explanation is a litmus test for not bothering about security, even today. It's worth asking for explanation, or at least a comment that all the buildbot contributors I've talked to have put a lot of effort into security configuration. This is a valid point. I think that Cameron and I may have had significantly different assumptions about the environment being discussed here. I may have brought some assumptions about the build farm here that don't actually apply to the way Python does it. To sum up what I believe is now the consensus from this thread: Anyone setting up a buildslave should take care to invoke the build in an environment where an out-of-control buildbot, potentially executing arbitrarily horrible and/or malicious code, should not damage anything. Builders should always be isolated from valuable resources, although the specific mechanism of isolation may differ. A virtual machine is a good default, but may not be sufficient; other tools for cutting of the builder from the outside world would be chroot jails, solaris zones, etc. Code runs differently as privileged vs. unprivileged users. Therefore builders should be set up in both configurations, running the full test suite, to ensure that all code runs as expected in both configurations. Some tests, as the start of this thread indicates, must have some special logic to make sure they do or do not run, or run differently, in privileged vs. unprivileged configurations, but generally speaking most things should work in both places. Access to root my provide access to slightly surprising resources, even within a VM (such as the ability to send spoofed IP packets, change the MAC address of even virtual ethernet cards, etc), and administrators should be aware that this is the case when configuring the host environment for a run-as-root builder. You don't want to end up with a compromised test VM that can snoop on your network. Have I left anything out? :-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail.com I've created an issue with a patch for a dedicated page in the devguide on running a build slave[1]. I've included the information from this thread on that page. I realize that the thread still has some juice in it, so the info I copied from this thread is likely incomplete and/or too much detail, but I wanted to get the devguide page rolling. -eric [1] http://bugs.python.org/issue13124 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Packaging and binary distributions for Python 3.3
I see that the Packaging documentation is now more complete (at least at docs.python.org) - I don't know if it's deemed fully complete yet, but I scanned the documentation and Installing Python Projects looks pretty much converted (and very good!!), but Distributing Python Projects still has quite a lot of distutils-related text in, and I need to read more deeply to understand if that's because it remains unchanged, or if it is still to be updated. But one thing struck me - the Installing Python Projects document talks about source distributions, but not much about binary distributions. On Windows, binary distributions are significantly more important than on Unix, because not all users have easy access to a compiler, and more importantly, C library dependencies can be difficult to build, hard to set up, and generally a pain to deal with. The traditional solution was always bdist_wininst installers, and with the advent of setuptools binary eggs started to become common. I've noticed that since pip became more widely used, with its focus on source builds, binary eggs seemed to fade away somewhat. I don't know what format packaging favours. The problem when Python 3.3 comes out is that bdist_wininst/bdist_msi installers do not interact well with pysetup. And if native virtual environment support becomes part of Python 3.3, they won't work well there either (they don't deal well with today's virtualenv, for that matter). So there will be a need for a pysetup-friendly binary format. I assume that the egg format will fill this role - or is that not the case? What is the current thinking on binary distribution formats for Python 3.3? The main reason I am asking is that I would like to write an article (or maybe a series of articles) for Python Insider, introducing the new packaging facilities from the point of view of an end user with straightforward needs (whether a package user just looking to manage a set of installed packages, or a module author who just wants to publish his code in a form that satisfies as many people as possible). What I'd hope to do is, as well as showing people all the nice things they can expect to see in Python 3.3, to also start package authors thinking about what they need to do to support their users under the new system. If we get the message out early, and make people aware of the benefits of the new end user tools, then I'm hoping more authors will see the advantage of switching to the new format rather than just sticking with bdist_xxx because it's always worked. I suspect I should (re-)join the distutils SIG and take this discussion there. But honestly, I'm not sure I have the time - the traffic was always fairly high, and the number of relevant posts for a casual observer was quite low. So even if that's the right place to go, some pointers to some high spots to get me up to speed on the current state of affairs would help. Thanks, Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PyUnicode_KIND changed
After discussion with several people, I changed PyUnicode_KIND to have values of 1,2,4, respectively, thus reflecting the element size of the string numerically. As a consequence, the PyUnicode_CHARACTER_SIZE and PyUnicode_KIND_SIZE macros are now gone. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyUnicode_KIND changed
On Fri, Oct 7, 2011 at 12:02 PM, Martin v. Löwis mar...@v.loewis.de wrote: After discussion with several people, I changed PyUnicode_KIND to have values of 1,2,4, respectively, thus reflecting the element size of the string numerically. Hah! I suggested this when first reviewing the PEP. :-) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SimpleHTTPServer slashdot (Was: Python Core Tools)
On Fri, Oct 7, 2011 at 3:57 PM, anatoly techtonik techto...@gmail.com wrote: On Sun, Oct 2, 2011 at 3:17 PM, Maciej Fijalkowski fij...@gmail.com wrote: On Sun, Oct 2, 2011 at 8:05 AM, Maciej Fijalkowski fij...@gmail.com wrote: On Sun, Oct 2, 2011 at 5:02 AM, anatoly techtonik techto...@gmail.com wrote: Hello, I've stumbled upon Dave Beazley's article [1] about trying ancient GIL removal patch at http://dabeaz.blogspot.com/2011/08/inside-look-at-gil-removal-patch-of.html and looking at the output of Python dis module thought that it would be cool if there were tools to inspect, explain and play with Python bytecode. Little visual assembler, that shows bytecode and disassembly side by side and annotates the listing with useful hints (like interpreter code optimization decisions). That will greatly help many new people understand how Python works and explain complicated stuff like GIL and stackless by copy/pasting pictures from there. PyPy has a tool named 'jitviewer' [2] that may be that I am looking for, but the demo is offline. I put demo back online. It's just that SimpleHTTPServer doesn't quite survive slashdot effect. Where do I fill a bug report :) http://bugs.python.org don't have a reproducible workload, it has something to do with stalling on sendall though (Something!) Is the demo address is still http://wyvern.cs.uni-duesseldorf.de:5000/ ? Still can't connect. =| Restarted -- anatoly t. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyUnicode_KIND changed
Le vendredi 7 octobre 2011 21:02:00, Martin v. Löwis a écrit : After discussion with several people, I changed PyUnicode_KIND to have values of 1,2,4, respectively, thus reflecting the element size of the string numerically. You may rename it to character size (char_size) ;-) Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New stringbench benchmark results
Le jeudi 6 octobre 2011 02:06:30, Victor Stinner a écrit :
The rfind case is really strange: the code between Python 3.2 and 3.3 is
exactly the same. Even in Python 3.2: rfind looks twice faster than find:
(AB*300+C).find(BC) (*1000) : 1.21
(C+AB*300).rfind(CA) (*1000) : 0.57
It looks to be a gcc bug: using attached patch (written by Antoine), str.find()
is a little bit faster. With the patch, the function does the same memory
access, but it generates a different machine code.
I don't know exactly the difference yet, but it may be related to the CMOVNE
instruction (which looks to be slower than a classical conditional jump, JNE).
Victor
diff --git a/Objects/stringlib/fastsearch.h b/Objects/stringlib/fastsearch.h
--- a/Objects/stringlib/fastsearch.h
+++ b/Objects/stringlib/fastsearch.h
@@ -76,6 +76,8 @@ FASTSEARCH(const STRINGLIB_CHAR* s, Py_s
mask = 0;
if (mode != FAST_RSEARCH) {
+const STRINGLIB_CHAR *ss = s + m - 1;
+const STRINGLIB_CHAR *pp = p + m - 1;
/* create compressed boyer-moore delta 1 table */
@@ -90,7 +92,7 @@ FASTSEARCH(const STRINGLIB_CHAR* s, Py_s
for (i = 0; i = w; i++) {
/* note: using mlast in the skip path slows things down on x86 */
-if (s[i+m-1] == p[m-1]) {
+if (ss[i] == pp[0]) {
/* candidate match */
for (j = 0; j mlast; j++)
if (s[i+j] != p[j])
@@ -106,13 +108,13 @@ FASTSEARCH(const STRINGLIB_CHAR* s, Py_s
continue;
}
/* miss: check if next character is part of pattern */
-if (!STRINGLIB_BLOOM(mask, s[i+m]))
+if (!STRINGLIB_BLOOM(mask, ss[i+1]))
i = i + m;
else
i = i + skip;
} else {
/* skip: check if next character is part of pattern */
-if (!STRINGLIB_BLOOM(mask, s[i+m]))
+if (!STRINGLIB_BLOOM(mask, ss[i+1]))
i = i + m;
}
}___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Disabling cyclic GC in timeit module
Hi Can we disable by default disabling the cyclic gc in timeit module? Often posts on pypy-dev or on pypy bugs contain usage of timeit module which might change the performance significantly. A good example is json benchmarks - you would rather not disable cyclic GC when running a web app, so encoding/decoding json in benchmark with the cyclic GC disabled does not make sense. What do you think? Cheers, fijal ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Disabling cyclic GC in timeit module
On Fri, Oct 7, 2011 at 4:50 PM, Maciej Fijalkowski fij...@gmail.com wrote: Hi Can we disable by default disabling the cyclic gc in timeit module? Often posts on pypy-dev or on pypy bugs contain usage of timeit module which might change the performance significantly. A good example is json benchmarks - you would rather not disable cyclic GC when running a web app, so encoding/decoding json in benchmark with the cyclic GC disabled does not make sense. What do you think? No, it's disabled by default for a reason (to avoid irrelevant noise in microbenchmarks), and other cases don't trump those original use cases. A command line switch to leave it enabled would probably be reasonable, though. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Disabling cyclic GC in timeit module
On Fri, Oct 7, 2011 at 11:47 PM, Nick Coghlan ncogh...@gmail.com wrote: On Fri, Oct 7, 2011 at 4:50 PM, Maciej Fijalkowski fij...@gmail.com wrote: Hi Can we disable by default disabling the cyclic gc in timeit module? Often posts on pypy-dev or on pypy bugs contain usage of timeit module which might change the performance significantly. A good example is json benchmarks - you would rather not disable cyclic GC when running a web app, so encoding/decoding json in benchmark with the cyclic GC disabled does not make sense. What do you think? No, it's disabled by default for a reason (to avoid irrelevant noise in microbenchmarks), and other cases don't trump those original use cases. People don't use it only for microbenchmarks though. Also, you can't call noise a thing that adds something every now and then I think. Er. How is disabling the GC for microbenchmarks any good by the way? Cheers, fijal ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] check for PyUnicode_READY look backwards
On 10/7/2011 10:06 AM, Nick Coghlan wrote: On Fri, Oct 7, 2011 at 9:21 AM, Martin v. Löwismar...@v.loewis.de wrote: if (!PyUnicode_READY(foo)) is not better, also because of PyUnicode_IS_READY(foo). I prefer PyUnicode_IS_READY(foo) 0 over PyUnicode_IS_READY(foo) == -1. Ok, so feel free to replace all == -1 tests with 0 tests as well. I'll point out that the test for -1 is also widespread in Python, e.g. when checking return values from PyObject_SetAttrString, BaseException_init, PyThread_create_key, PyObject_DelAttrString, etc. FWIW, I don't mind whether it's 0 or == -1, so long as there's a comparison there to kick my brain out of Python boolean logic mode. Is there any speed difference (on common x86/64 processors and compilers)? I would expect that ' 0' should be optimized to just check the sign bit and 'if n 0' to 'load n; jump-non-negative'. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
On 10/7/2011 6:18 AM, Glyph wrote: To sum up what I believe is now the consensus from this thread: 1. Anyone setting up a buildslave should take care to invoke the build in an environment where an out-of-control buildbot, potentially executing arbitrarily horrible and/or malicious code, should not damage anything. Builders should always be isolated from valuable resources, although the specific mechanism of isolation may differ. A virtual machine is a good default, but may not be sufficient; other tools for cutting of the builder from the outside world would be chroot jails, solaris zones, etc. 2. Code runs differently as privileged vs. unprivileged users. My particular concern with testing as an unprivileged user comes from experience with too many (commercial, post-XP) Windows programs that only run correctly as admin (without an obvious good reason). Therefore builders should be set up in both configurations, running the full test suite, to ensure that all code runs as expected in both configurations. Some tests, as the start of this thread indicates, must have some special logic to make sure they do or do not run, or run differently, in privileged vs. unprivileged configurations, but generally speaking most things should work in both places. 3. Access to root my provide access to slightly surprising resources, even within a VM (such as the ability to send spoofed IP packets, change the MAC address of even virtual ethernet cards, etc), and administrators should be aware that this is the case when configuring the host environment for a run-as-root builder. You don't want to end up with a compromised test VM that can snoop on your network. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Disabling cyclic GC in timeit module
On Sat, 8 Oct 2011 00:13:40 +0200 Maciej Fijalkowski fij...@gmail.com wrote: On Fri, Oct 7, 2011 at 11:47 PM, Nick Coghlan ncogh...@gmail.com wrote: On Fri, Oct 7, 2011 at 4:50 PM, Maciej Fijalkowski fij...@gmail.com wrote: Hi Can we disable by default disabling the cyclic gc in timeit module? Often posts on pypy-dev or on pypy bugs contain usage of timeit module which might change the performance significantly. A good example is json benchmarks - you would rather not disable cyclic GC when running a web app, so encoding/decoding json in benchmark with the cyclic GC disabled does not make sense. What do you think? No, it's disabled by default for a reason (to avoid irrelevant noise in microbenchmarks), and other cases don't trump those original use cases. People don't use it only for microbenchmarks though. Also, you can't call noise a thing that adds something every now and then I think. Er. How is disabling the GC for microbenchmarks any good by the way? In CPython, looking for reference cycles is a parasitic task that interferes with what you are trying to measure. It is not critical in any way, and you can schedule it much less often if it takes too much CPU, without any really adverse consequences. timeit takes the safe way and disables it completely. In PyPy, it doesn't seem gc.disable() should do anything, since you'd lose all automatic memory management if the GC was disabled. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
On 08/10/2011 00:19, Terry Reedy wrote: On 10/7/2011 6:18 AM, Glyph wrote: To sum up what I believe is now the consensus from this thread: 1. Anyone setting up a buildslave should take care to invoke the build in an environment where an out-of-control buildbot, potentially executing arbitrarily horrible and/or malicious code, should not damage anything. Builders should always be isolated from valuable resources, although the specific mechanism of isolation may differ. A virtual machine is a good default, but may not be sufficient; other tools for cutting of the builder from the outside world would be chroot jails, solaris zones, etc. 2. Code runs differently as privileged vs. unprivileged users. My particular concern with testing as an unprivileged user comes from experience with too many (commercial, post-XP) Windows programs that only run correctly as admin (without an obvious good reason). It would seem that for this use case it is more important that all tests pass when run as a *non-admin* user. Michael Therefore builders should be set up in both configurations, running the full test suite, to ensure that all code runs as expected in both configurations. Some tests, as the start of this thread indicates, must have some special logic to make sure they do or do not run, or run differently, in privileged vs. unprivileged configurations, but generally speaking most things should work in both places. 3. Access to root my provide access to slightly surprising resources, even within a VM (such as the ability to send spoofed IP packets, change the MAC address of even virtual ethernet cards, etc), and administrators should be aware that this is the case when configuring the host environment for a run-as-root builder. You don't want to end up with a compromised test VM that can snoop on your network. -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Disabling cyclic GC in timeit module
On Sat, Oct 8, 2011 at 1:47 AM, Antoine Pitrou solip...@pitrou.net wrote: On Sat, 8 Oct 2011 00:13:40 +0200 Maciej Fijalkowski fij...@gmail.com wrote: On Fri, Oct 7, 2011 at 11:47 PM, Nick Coghlan ncogh...@gmail.com wrote: On Fri, Oct 7, 2011 at 4:50 PM, Maciej Fijalkowski fij...@gmail.com wrote: Hi Can we disable by default disabling the cyclic gc in timeit module? Often posts on pypy-dev or on pypy bugs contain usage of timeit module which might change the performance significantly. A good example is json benchmarks - you would rather not disable cyclic GC when running a web app, so encoding/decoding json in benchmark with the cyclic GC disabled does not make sense. What do you think? No, it's disabled by default for a reason (to avoid irrelevant noise in microbenchmarks), and other cases don't trump those original use cases. People don't use it only for microbenchmarks though. Also, you can't call noise a thing that adds something every now and then I think. Er. How is disabling the GC for microbenchmarks any good by the way? In CPython, looking for reference cycles is a parasitic task that interferes with what you are trying to measure. It is not critical in any way, and you can schedule it much less often if it takes too much CPU, without any really adverse consequences. timeit takes the safe way and disables it completely. In PyPy, it doesn't seem gc.disable() should do anything, since you'd lose all automatic memory management if the GC was disabled. it disables finalizers but this is besides the point. the point is that people use timeit module to compute absolute time it takes for CPython to do things, among other things comparing it to PyPy. While I do agree that in microbenchmarks you don't loose much by just disabling it, it does affect larger applications. So answering the question like how much time will take json encoding in my application should take cyclic GC time into account. Cheers, fijal ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Disabling cyclic GC in timeit module
In CPython, looking for reference cycles is a parasitic task that interferes with what you are trying to measure. It is not critical in any way, and you can schedule it much less often if it takes too much CPU, without any really adverse consequences. timeit takes the safe way and disables it completely. In PyPy, it doesn't seem gc.disable() should do anything, since you'd lose all automatic memory management if the GC was disabled. it disables finalizers but this is besides the point. the point is that people use timeit module to compute absolute time it takes for CPython to do things, among other things comparing it to PyPy. While I do agree that in microbenchmarks you don't loose much by just disabling it, it does affect larger applications. So answering the question like how much time will take json encoding in my application should take cyclic GC time into account. If you are only measuring json encoding of a few select pieces of data then it's a microbenchmark. If you are measuring the whole application (or a significant part of it) then I'm not sure timeit is the right tool for that. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Disabling cyclic GC in timeit module
On Sat, Oct 8, 2011 at 2:18 AM, Antoine Pitrou solip...@pitrou.net wrote: In CPython, looking for reference cycles is a parasitic task that interferes with what you are trying to measure. It is not critical in any way, and you can schedule it much less often if it takes too much CPU, without any really adverse consequences. timeit takes the safe way and disables it completely. In PyPy, it doesn't seem gc.disable() should do anything, since you'd lose all automatic memory management if the GC was disabled. it disables finalizers but this is besides the point. the point is that people use timeit module to compute absolute time it takes for CPython to do things, among other things comparing it to PyPy. While I do agree that in microbenchmarks you don't loose much by just disabling it, it does affect larger applications. So answering the question like how much time will take json encoding in my application should take cyclic GC time into account. If you are only measuring json encoding of a few select pieces of data then it's a microbenchmark. If you are measuring the whole application (or a significant part of it) then I'm not sure timeit is the right tool for that. Regards Antoine. When you're measuring how much time it takes to encode json, this is a microbenchmark and yet the time that timeit gives you is misleading, because it'll take different amount of time in your application. I guess my proposition would be to not disable gc by default and disable it when requested, but well, I guess I'll give up given the strong push against it. Cheers, fijal ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Disabling cyclic GC in timeit module
Antoine Pitrou wrote: In CPython, looking for reference cycles is a parasitic task that interferes with what you are trying to measure. It is not critical in any way, and you can schedule it much less often if it takes too much CPU, without any really adverse consequences. timeit takes the safe way and disables it completely. In PyPy, it doesn't seem gc.disable() should do anything, since you'd lose all automatic memory management if the GC was disabled. it disables finalizers but this is besides the point. the point is that people use timeit module to compute absolute time it takes for CPython to do things, among other things comparing it to PyPy. While I do agree that in microbenchmarks you don't loose much by just disabling it, it does affect larger applications. So answering the question like how much time will take json encoding in my application should take cyclic GC time into account. If you are only measuring json encoding of a few select pieces of data then it's a microbenchmark. If you are measuring the whole application (or a significant part of it) then I'm not sure timeit is the right tool for that. Perhaps timeit should grow a macro-benchmark tool too? I find myself often using timeit to time macro-benchmarks simply because it's more convenient at the interactive interpreter than the alternatives. Something like this idea perhaps? http://preshing.com/20110924/timing-your-code-using-pythons-with-statement -- Steven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
On 08Oct2011 01:13, Michael Foord fuzzy...@voidspace.org.uk wrote: | On 08/10/2011 00:19, Terry Reedy wrote: | On 10/7/2011 6:18 AM, Glyph wrote: | | To sum up what I believe is now the consensus from this thread: | | 1. Anyone setting up a buildslave should take care to invoke the build | in an environment where an out-of-control buildbot, potentially | executing arbitrarily horrible and/or malicious code, should not | damage anything. Builders should always be isolated from valuable | resources, although the specific mechanism of isolation may differ. | A virtual machine is a good default, but may not be sufficient; | other tools for cutting of the builder from the outside world would | be chroot jails, solaris zones, etc. | 2. Code runs differently as privileged vs. unprivileged users. | | My particular concern with testing as an unprivileged user comes | from experience with too many (commercial, post-XP) Windows | programs that only run correctly as admin (without an obvious good | reason). | | It would seem that for this use case it is more important that all | tests pass when run as a *non-admin* user. I'm pretty sure that's what Terry meant; if these apps were tested non-admin they wouldn't need to run as admin (without an obvious good reason. Cheers, -- Cameron Simpson c...@zip.com.au DoD#743 http://www.cskk.ezoshosting.com/cs/ It is easier to optimize correct code than to correct optimized code. - Bill Harlan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Disabling cyclic GC in timeit module
Perhaps timeit should grow a macro-benchmark tool too? I find myself often using timeit to time macro-benchmarks simply because it's more convenient at the interactive interpreter than the alternatives. Something like this idea perhaps? http://preshing.com/20110924/timing-your-code-using-pythons-with-statement I have essentially the same snippet (with the addition of being able to provide names for timers, thus allowing to have several executing in the code and knowing which is which) lying in my toolbox for a long time now, and I find it very useful. There's also an alternative approach, having a decorator that marks a function for benchmarking. David Beazley has one good example of this here: http://www.dabeaz.com/python3io/timethis.py Eli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
