[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)
On Wed, Jan 05, 2022 at 01:59:21PM -0800, Trent Nelson wrote: > > I did some research on this a few years back. I was curious what sort > of "max reference counts" were encountered in the wild, in long-running > real life programs. For the same reason: I wanted to get some insight > into how many unused bits could possibly be repurposed for future > shenanigans (I had PyParallel* in the mind at the time). > > I added some logic to capture* the max reference counts of the None, > True, and Zero objects (in a trace callback), then ran a really long > simulation program of a client's (it ran for about 5-6 hours). The > results were as follows: > > MaxNoneRefCount 9,364,132 > MaxTrueRefCount 204,215 > MaxZeroRefCount36,784 Just double-checked my results, there were a handful of runs with higher counts: MaxNoneRefCount 59,834,444 MaxTrueRefCount 1,072,467 MaxZeroRefCount 3,460,921 Regards, Trent. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UJPQVBF5I4PGZTBX3EOVLLCAOQVNHVGZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)
On Wed, Dec 15, 2021 at 02:57:46PM -0800, Guido van Rossum wrote: > On Wed, Dec 15, 2021 at 6:04 AM Antoine Pitrou wrote: > > > On Wed, 15 Dec 2021 14:13:03 +0100 > > Antoine Pitrou wrote: > > > > > Did you try to take into account the envisioned project for adding a > > > "complete" GC and removing the GIL? > > > > Sorry, I was misremembering the details. Sam Gross' proposal > > (posted here on 07/10/2021) doesn't switch to a "complete GC", but it > > changes reference counting to a more sophisticated scheme (which > > includes immortalization of objects): > > > > > > https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlejd9l0/edit > > > > A note about this: Sam's immortalization covers exactly the objects that > Eric is planning to move into the interpreter state struct: "such as > interned strings, small integers, statically allocated PyTypeObjects, and > the True, False, and None objects". (Well, he says "such as" but I think so > does Eric. :-) > > Sam's approach is to use the lower bit of the ob_refcnt field to indicate > immortal objects. This would not work given the stable ABI (which has > macros that directly increment and decrement the ob_refcnt field). In fact, > I think that Sam's work doesn't preserve the stable ABI at all. However, > setting a very high bit (the bit just below the sign bit) would probably > work. Say we're using 32 bits. We use the value 0x_6000_ as the initial > refcount for immortal objects. The stable ABI will sometimes increment > this, sometimes decrement it. But as long as the imbalance is less than > 0x_2000_, the refcount will remain in the inclusive range [ > 0x_4000_ , 0x_7FFF_ ] and we can test for immortality by testing a > single bit: > > if (o->ob_refcnt & 0x_4000_) > > I don't know how long that would take, but I suspect that a program that > just increments the refcount relentlessly would have to run for hours > before hitting this range. On a 64-bit machine the same approach would > require years to run before a refcount would exceed the maximum allowable > imbalance. (These estimates are from Mark Shannon.) I did some research on this a few years back. I was curious what sort of "max reference counts" were encountered in the wild, in long-running real life programs. For the same reason: I wanted to get some insight into how many unused bits could possibly be repurposed for future shenanigans (I had PyParallel* in the mind at the time). I added some logic to capture* the max reference counts of the None, True, and Zero objects (in a trace callback), then ran a really long simulation program of a client's (it ran for about 5-6 hours). The results were as follows: MaxNoneRefCount 9,364,132 MaxTrueRefCount 204,215 MaxZeroRefCount36,784 I thought that was pretty interesting. Potentially many, many upper bits for the taking. The code also had some logic that would int 3 as soon as a 32-bit refcnt overflowed, and that never hit either (obviously, based on the numbers above). I also failed to come up with real-life code that would result in a Python object having a reference count higher than None's refcnt, but that may have just been from lack of creativity. Just thought I'd share. Regards, Trent. [*] 1: https://github.com/pyparallel/pyparallel [*] 2: https://github.com/tpn/tracer/blob/master/PythonTracer/PythonTracer.h#L690 ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/U2WHO5LYMBF6A6AFM36HOQQCNVGLXG6M/ Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-Dev] Discussion related to memory leaks requested
Gflags/umdh is pretty useful on Windows, I used it to track down a few quirky PyParallel memory leaks. Steps: 1. Enable global flags: gflags –i python.exe +ust 2. Launch Python. 3. Enable the umdh tracer: umdh –p: -f:d1.log 4. Kill it after a short run. 5. Re-launch Python. 6. Enable it again: umdh –p: -f:d2.log 7. Let it run for longer – long enough to make sure it’s leaking memory. 8. Kill it, then generate a dump file: umdh –d d1.log d2.log > dump.txt (Those steps were pretty specific to my particular situation, but it should at least be a reasonable starting point for what to google to find out more.) Here are two sample outputs that pin-pointed the exact leak path: + 49116 ( 49116 - 0) 6 allocsBackTrace9763CA0 + 6 ( 6 - 0)BackTrace9763CA0allocations ntdll!RtlpCallInterceptRoutine+40 ntdll!RtlAllocateHeap+79846 SQLSRV32!SQLAllocateMemory+26 markSQLSRV32!SQLAllocConnect+F6 SQLSRV32!SQLAllocHandle+83 ODBC32!RetcodeDriverInit+2D9 ODBC32!SQLInternalDriverConnectW+2F ODBC32!CDispenser::CreateResource+DB comsvcs!CHolder::SafeDispenserDriver::CreateResource+43 comsvcs!CHolder::AllocResource+24D ODBC32!CDispenser::TryAllocResource+6E ODBC32!CDispenser::GetActiveConnection+72 ODBC32!SQLDriverConnectW+9D4 pyodbc!Connect+14F (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\connection.cpp, 85) pyodbc!Connection_New+CD (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\connection.cpp, 166) pyodbc!mod_connect+579 (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\pyodbcmodule.cpp, 378) python33!PyCFunction_Call+F3 (c:\users\trent\home\src\pyparallel\objects\methodobject.c, 84) python33!call_function+371 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4130) python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745) python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4219) python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4152) python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745) python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4219) python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4152) python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745) python33!PyEval_EvalCodeEx+B4D (c:\users\trent\home\src\pyparallel\python\ceval.c, 3500) python33!function_call+1BB (c:\users\trent\home\src\pyparallel\objects\funcobject.c, 639) python33!PyObject_Call+7C (c:\users\trent\home\src\pyparallel\objects\abstract.c, 2036) python33!method_call+F9 (c:\users\trent\home\src\pyparallel\objects\classobject.c, 353) python33!PyObject_Call+7C (c:\users\trent\home\src\pyparallel\objects\abstract.c, 2036) python33!PyEval_CallObjectWithKeywords+16C (c:\users\trent\home\src\pyparallel\python\ceval.c, 4011) python33!PxSocket_IOLoop+1249 (c:\users\trent\home\src\pyparallel\python\pyparallel.c, 9128) + 48432 ( 48432 - 0) 6 allocsBackTrace97635E0 + 6 ( 6 - 0)BackTrace97635E0allocations ntdll!RtlpCallInterceptRoutine+40 ntdll!RtlAllocateHeap+79846 SQLSRV32!SQLAllocateMemory+26 SQLSRV32!SQLAllocConnect+4D SQLSRV32!SQLAllocHandle+83 ODBC32!RetcodeDriverInit+2D9 ODBC32!SQLInternalDriverConnectW+2F ODBC32!CDispenser::CreateResource+DB comsvcs!CHolder::SafeDispenserDriver::CreateResource+43 comsvcs!CHolder::AllocResource+24D ODBC32!CDispenser::TryAllocResource+6E ODBC32!CDispenser::GetActiveConnection+72 ODBC32!SQLDriverConnectW+9D4 pyodbc!Connect+14F (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\connection.cpp, 85) pyodbc!Connection_New+CD (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\connection.cpp, 166) pyodbc!mod_connect+579 (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\pyodbcmodule.cpp, 378) python33!PyCFunction_Call+F3 (c:\users\trent\home\src\pyparallel\objects\methodobject.c, 84) python33!call_function+371 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4130) python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745) python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4219) python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4152) python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745) python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4219) python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4152) python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745) python33!PyEval_EvalCodeEx+B4D (c:\users\trent\home\src\pyparallel\python\ceval.c, 3500) python33!function_call+1BB
Re: [Python-Dev] Yet another "A better story for multi-core Python" comment
On Wed, Sep 09, 2015 at 04:52:39PM -0400, Gary Robinson wrote: > I’m going to seriously consider installing Windows or using a > dedicated hosted windows box next time I have this problem so that I > can try your solution. It does seem pretty ideal, although the STM > branch of PyPy (using http://codespeak.net/execnet/ to access SciPy) > might also work at this point. I'm not sure how up-to-date this is: http://pypy.readthedocs.org/en/latest/stm.html But it sounds like there's a 1.5GB memory limit (or maybe 2.5GB now, I just peaked at core.h linked in that page) and a 4-core segment limit. PyParallel has no memory limit (although it actually does have support for throttling back memory pressure by not accepting new connections when the system hits 90% physical memory used) and no core limit, and it scales linearly with cores+concurrency. PyPy-STM and PyParallel are both pretty bleeding edge and experimental though so I'm sure we both crash as much as each other when exercised outside of our comfort zones :-) I haven't tried getting the SciPy stack running with PyParallel yet. Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Yet another "A better story for multi-core Python" comment
On Tue, Sep 08, 2015 at 10:12:37AM -0400, Gary Robinson wrote: > There was a huge data structure that all the analysis needed to > access. Using a database would have slowed things down too much. > Ideally, I needed to access this same structure from many cores at > once. On a Power8 system, for example, with its larger number of > cores, performance may well have been good enough for production. In > any case, my experimentation and prototyping would have gone more > quickly with more cores. > > But this data structure was simply too big. Replicating it in > different processes used memory far too quickly and was the limiting > factor on the number of cores I could use. (I could fork with the big > data structure already in memory, but copy-on-write issues due to > reference counting caused multiple copies to exist anyway.) This problem is *exactly* the type of thing that PyParallel excels at, just FYI. PyParallel can load large, complex data structures now, and then access them freely from within multiple threads. I'd recommended taking a look at the "instantaneous Wikipedia search server" example as a start: https://github.com/pyparallel/pyparallel/blob/branches/3.3-px/examples/wiki/wiki.py That loads trie with 27 million entries, creates ~27.1 million PyObjects, loads a huge NumPy array, and has a WSS of ~11GB. I've actually got a new version in development that loads 6 tries of the most frequent terms for character lengths 1-6. Once everything is loaded, the data structures can be accessed for free in parallel threads. There are more details regarding how this is achieved on the landing page: https://github.com/pyparallel/pyparallel I've done a couple of consultancy projects now that were very data science oriented (with huge data sets), so I really gained an appreciation for how common the situation you describe is. It is probably the best demonstration of PyParallel's strengths. > Gary Robinson gary...@me.com http://www.garyrobinson.net Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Yet another "A better story for multi-core Python" comment
On Wed, Sep 09, 2015 at 01:43:19PM -0700, Ethan Furman wrote: > On 09/09/2015 01:33 PM, Trent Nelson wrote: > > >This problem is *exactly* the type of thing that PyParallel excels at [...] > > Sorry if I missed it, but is PyParallel still Windows only? Yeah, still Windows only. Still based off 3.3.5. I'm hoping to rebase off 3.5 after its tagged and get it into a state where it can at least build on POSIX (i.e. stub enough functions such that it'll compile). That's going to be a lot of work though, would love to get some help with it. Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PyParallel update
[CC'ing python-dev@ for those that are curious; please drop and keep follow-up discussion to python-ideas@] Hi folks, I've made a lot of progress on PyParallel since the PyCon dev summit (https://speakerdeck.com/trent/pyparallel-pycon-2015-language-summit); I fixed the outstanding breakage with generators, exceptions and whatnot. I got the "instantaneous Wiki search server" working[1] and implemented the entire TechEmpower Frameworks Benchmark Suite[2], including a PyParallel-friendly pyodbc module, allowing database connections and querying in parallel. [1]: https://github.com/pyparallel/pyparallel/blob/branches/3.3-px/examples/wiki/wiki.py [2]: https://github.com/pyparallel/pyparallel/blob/branches/3.3-px/examples/tefb/tefb.py I set up a landing page for the project: http://pyparallel.org And there was some good discussion on reddit earlier this week: https://www.reddit.com/r/programming/comments/3jhv80/pyparallel_an_experimental_proofofconcept_fork_of/ I've put together some documentation on the project, its aims, and the key parts of the solution regarding the parallelism through simple client/server paradigms. This documentation is available directly on the github landing page for the project: https://github.com/pyparallel/pyparallel Writing that documentation forced me to formalize (or at least commit) to the restrictions/trade-offs that PyParallel would introduce, and I'm pretty happy I was basically able to boil it down into a single rule: Don't persist parallel objects. That keeps the mental model very simple. You don't need to worry about locking or ownership or races or anything like that. Just don't persist parallel objects, that's the only thing you have to remember. It's actually really easy to convert existing C code or Python code into something that is suitable for calling from within a parallel callback by just ensuring that rule isn't violated. It took about four hours to figure out how NumPy allocated stuff and add in the necessary PyParallel-aware tweaks, and not that much longer for pyodbc. (Most stuff "just works", though.) (The ABI changes would mean this is a Python 4.x type of thing; there are fancy ways we could avoid ABI changes and get this working on Python 3.x, but, eh, I like the 4.x target. It's realistic.) The other thing that clicked is that asyncio and PyParallel would actually work really well together for exploiting client-driven parallelism (PyParallel really is only suited to server-oriented parallelism at the moment, i.e. serving HTTP requests in parallel). With asyncio, though, you could keep the main-thread/single-thread client-drives-computation paradigm, but have it actually dispatch work to parallel.server() objects behind the scenes. For example, in order to process all files in a directory in parallel, asyncio would request a directory listing (i.e. issue a GET /) which the PyParallel HTTP server would return, it would then create non-blocking client connections to the same server and invoke whatever HTTP method is desired to do the file processing. You can either choose to write the new results from within the parallel context (which could then be accessed as normal files via HTTP), or you could have PyParallel return json/bytes, which could then be aggregated by asyncio. Everything is within the same process, so you get all the benefits that provides (free access to anything within scope, like large data structures, from within parallel contexts). You can synchronously call back into the main thread from a parallel thread, too, if you wanted to update a complex data structure directly. The other interesting thing that documentation highlights is the advantage of the split brain "main thread vs parallel thread" GC and non-GC allocators. I'm not sure if I've ever extolled the virtue of such an approach on paper or in e-mail. It's pretty neat though and allows us to avoid a whole raft of problems that need to be solved when you have a single GC/memory model. Next steps: once 3.5 is tagged, I'm going to bite the bullet and rebase. That'll require a bit of churn, so if there's enough interest from others, I figured we'd use the opportunity to at least get it building again on POSIX (Linux/OSX/FreeBSD). From there people can start implementing the missing bits for implementing the parallel machinery behind the scenes. The parallel interpreter thread changes I made are platform agnostic, the implementation just happens to be on Windows at the moment; don't let the Windows-only thing detract from what's actually being pitched: a (working, demonstrably-performant) solution to "Python's GIL problem". Regards, Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [python-committers] Can we clean up the buildbots please?
On Fri, May 22, 2015 at 10:06:48PM +, Steve Dower wrote: The Windows 7 buildbots are failing on test_asdl_parser, but I have no idea why – the test works for me just fine. Yury and Benjamin made the most recent changes to Python.asdl, but I have no idea what effect they would have here, or why it’s Windows only. The WS2K3 machine needs a reboot – I pinged Trent about that months ago – and the XP one isn’t supported for 3.5. Gave it a little bit of love just then (haven't been able to access it for months as the main switch needed a reboot). There were like, 155 cl.exe processes wedged and a bunch of error reporting dialogs. Do we still support WS2K3? (Can I even install VS 2015 on that? I would have thought not.) Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [python-committers] Can we clean up the buildbots please?
On Fri, May 22, 2015 at 05:24:53PM -0700, Larry Hastings wrote: On 05/22/2015 05:11 PM, Trent Nelson wrote: Do we still support WS2K3? (Can I even install VS 2015 on that? I would have thought not.) According to PCbuild/readme.txt, no. It says: This directory is used to build CPython for Microsoft Windows NT version 6.0 or higher (Windows Vista, Windows Server 2008, or later) on 32 and 64 bit platforms. Ah, yeah, thought so. Pity, that box is probably the only one that hasn't had any form of hardware failure during its tenure ;-) Tried to get the W2K8 one back up on Monday when I had some remote hands but alas, no luck. Think it has balked HDDs or something. The Solaris 11 AMD64 one Solaris 10 SPARC ones are back up now though and I just cleared out their 700+ build backlogs, FWIW. Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bytes bytearray
On Tue, Jan 20, 2015 at 11:48:10AM +0200, Paul Sokolovsky wrote:
Hello,
On Tue, 20 Jan 2015 18:15:02 +1300
Greg Ewing greg.ew...@canterbury.ac.nz wrote:
Guido van Rossum wrote:
On Mon, Jan 19, 2015 at 11:43 AM, Paul Sokolovsky
pmis...@gmail.com mailto:pmis...@gmail.com wrote:
b.lower_inplace()
b.lower_i()
Please don't go there. The use cases are too rare.
And if you have such a use case, it's not too
hard to do
b[:] = b.lower()
The point of inplace operations (memoryview's, other stuff already in
Python) is to avoid unneeded memory allocation and copying. For 1Tb
bytearray with 1Tb of RAM, it will be very hard to do. (Ditto for 100K
bytearray with 150K RAM.)
You can just loop through the bytearray and assign elements. I use
something along the lines of this for PyParallel where I'm operating on
bytearrays that are backed by underlying socket buffers, where I don't
want to do any memory allocations/reallocations:
def toupper_bytes(data):
assert isinstance(data, bytearray)
a = ord('a')
z = ord('z')
for i in range(0, len(data)):
c = data[i]
if c = a and c = z:
data[i] = c - 32
Low overhead, mostly stays within the same ceval frame. Should be a
walk in the park for PyPy, Cython or Numba to optimize, too.
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Access control for buildbot
On Dec 10, 2014, at 6:56 AM, jacob toft pedersen ja...@luxion.com wrote: Hi there I was visiting you buildbot page for inspiration and found that i apparently have the option to force stop/start all your builds without any access control. You may want to put something to enforce access control? Nah, as far as I know, no-one has abused it, and it’s definitely useful when you need to legitimately use it. Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The pysandbox project is broken
On Sat, Nov 16, 2013 at 02:53:22AM -0800, Maciej Fijalkowski wrote: On Fri, Nov 15, 2013 at 6:56 PM, Trent Nelson tr...@snakebite.org wrote: On Tue, Nov 12, 2013 at 01:16:55PM -0800, Victor Stinner wrote: pysandbox cannot be used in practice To protect the untrusted namespace, pysandbox installs a lot of different protections. Because of all these protections, it becomes hard to write Python code. Basic features like del dict[key] are denied. Passing an object to a sandbox is not possible to sandbox, pysandbox is unable to proxify arbitary objects. For something more complex than evaluating 1+(2*3), pysandbox cannot be used in practice, because of all these protections. Individual protections cannot be disabled, all protections are required to get a secure sandbox. This sounds a lot like the work I initially did with PyParallel to try and intercept/prevent parallel threads mutating main-thread objects. I ended up arriving at a much better solution by just relying on memory protection; main thread pages are set read-only prior to parallel threads being able to run. If a parallel thread attempts to mutate a main thread object; a SEH is raised (SIGSEV on POSIX), which I catch in the ceval loop and convert into an exception. See slide 138 of this: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores-1 I'm wondering if this sort of an approach (which worked surprisingly well) could be leveraged to also provide a sandbox environment? The goals are the same: robust protection against mutation of memory allocated outside of the sandbox. (I'm purely talking about memory mutation; haven't thought about how that could be extended to prevent file system interaction as well.) Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com Trent, you should read the mail more carefully. Notably the same issues that make it impossible to create a sandbox make it impossible to create pyparaller really work. Being read-only is absolutely not enough - you can read some internal structures in inconsistent state that lead to crashes and/or very unexpected behavior even without modifying anything. What do you mean by inconsistent state? Like a dict half way through `a['foo'] = 'bar'`? That can't happen with PyParallel; parallel threads don't run when the main thread runs and vice versa. The main thread's memory (and internal object structure) will always be consistent by the time the parallel threads run. PS. We really did a lot of work analyzing how STM-pypy can lead to conflicts and/or inconsistent behavior. But you support free-threading though, right? As in, code that subclasses threading.Thread should be able to benefit from your STM work? I explicitly don't support free-threading. Your threading.Thread code will not magically run faster with PyParallel. You'll need to re-write your code using the parallel and async façade APIs I expose. On the plus side, I can completely control everything about the main thread and parallel thread execution environments; obviating the need to protect against internal inconsistencies by virtue of the fact that the main thread will always be in a consistent state when the parallel threads are running. (And it works really well in practice; I ported SimpleHTTPServer to use my new async stuff and it flies -- it'll automatically exploit all your cores if there is sufficient incoming load. Unexpected side-effect of my implementation is that code executing in parallel callbacks actually runs faster than normal single-threaded Python code; no need to do reference counting, GC, and the memory model is ridiculously cache and TLB friendly.) This is getting off-topic though and I don't want to hijack the sandbox thread. I was planning on sending an e-mail in a few days when the PyData video of my talk is live -- we can debate the merits of my parallel/async approach then :-) Cheers, fijal Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PyParallel: alternate async I/O and GIL removal
Hi folks, Video of the presentation I gave last weekend at PyData NYC regarding PyParallel just went live: https://vimeo.com/79539317 Slides are here: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores-1 The work was driven by the async I/O discussions around this time last year on python-ideas. That resulted in me sending this: http://markmail.org/thread/kh3qgjbydvxt3exw#query:+page:1+mid:arua62vllzugjy2v+state:results where I attempted to argue that there was a better way of doing async I/O on Windows than the status quo of single-threaded, non-blocking I/O with an event multiplex syscall. I wasn't successful in convincing anyone at the time; I had no code to back it up and I didn't articulate my plans for GIL removal at the time either (figuring the initial suggestion would be met with enough scepticism as is). So, in the video above, I spend a lot of time detailing how IOCP works on Windows, how it presents us with a better environment than UNIX for doing asynchronous I/O, and how it paired nicely with the other work I did on coming up with a way for multiple threads to execute simultaneously across all cores without introducing any speed penalties. I'm particularly interested to hear if the video/slides helped UNIX-centric people gain a better understanding of how Windows does IOCP and why it would be preferable when doing async I/O. The reverse is also true: if you still think single-threaded, non- blocking synchronous I/O via kqueue/epoll is better than the approach afforded by IOCP, I'm interested in hearing why. As crazy as it sounds, my long term goal would be to try and influence Linux and BSD kernels to implement thread-agnostic I/O support such that an IOCP-like mechanism could be exposed; Solaris and AIX already do this via event ports and AIX's verbatim copy of Windows' IOCP API. (There is some promising work already being done on Linux; see recent MegaPipe paper for an example.) Regards, Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The pysandbox project is broken
On Tue, Nov 12, 2013 at 01:16:55PM -0800, Victor Stinner wrote: pysandbox cannot be used in practice To protect the untrusted namespace, pysandbox installs a lot of different protections. Because of all these protections, it becomes hard to write Python code. Basic features like del dict[key] are denied. Passing an object to a sandbox is not possible to sandbox, pysandbox is unable to proxify arbitary objects. For something more complex than evaluating 1+(2*3), pysandbox cannot be used in practice, because of all these protections. Individual protections cannot be disabled, all protections are required to get a secure sandbox. This sounds a lot like the work I initially did with PyParallel to try and intercept/prevent parallel threads mutating main-thread objects. I ended up arriving at a much better solution by just relying on memory protection; main thread pages are set read-only prior to parallel threads being able to run. If a parallel thread attempts to mutate a main thread object; a SEH is raised (SIGSEV on POSIX), which I catch in the ceval loop and convert into an exception. See slide 138 of this: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores-1 I'm wondering if this sort of an approach (which worked surprisingly well) could be leveraged to also provide a sandbox environment? The goals are the same: robust protection against mutation of memory allocated outside of the sandbox. (I'm purely talking about memory mutation; haven't thought about how that could be extended to prevent file system interaction as well.) Trent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The pysandbox project is broken
On Nov 15, 2013, at 12:34 PM, Victor Stinner wrote:
2013/11/15 Trent Nelson tr...@snakebite.org:
This sounds a lot like the work I initially did with PyParallel to
try and intercept/prevent parallel threads mutating main-thread
objects.
I ended up arriving at a much better solution by just relying on
memory protection; main thread pages are set read-only prior to
parallel threads being able to run. If a parallel thread attempts
to mutate a main thread object; a SEH is raised (SIGSEV on POSIX),
which I catch in the ceval loop and convert into an exception.
Read-only is not enough, an attack must not be able to read sensitive data.
Well you could remove both write *and* read perms from pages, such that you
would trap on read attempts too. What's an example of sensitive data that
you'd need to have residing in the same process that you also want to sandbox?
I was going to suggest something like:
with memory.protected:
htpasswd = open('htpasswd', 'r').read()
...
But then I couldn't think of why you'd persist the sensitive data past the
point you'd need it.
Protections of memory pages sound very low-level, so not very portable :-/
It's a pretty fundamental provision provided by operating systems; granted, the
interface differs (mprotect() versus VirtualProtect()), but the result is the
same.
How do you know fif SIGSEGV comes from a legal call (parallel thread
thing) or a real bug?
You don't, but it doesn't really matter. It'll be pretty obvious from looking
at the offending line of code in the exception whether it was a legitimate
memory protection error, or a bug in an extension module/CPython internals.
And having a ProtectionError bubble all the way back up to the top of the stack
with exact details about the offending frame/line could be considered a nicer
alternative to dumping core ;-) (Unless you happen to be in an `except: pass`
block.)
Victor
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] pyparallel and new memory API discussions...
The new memory API discussions (and PEP) warrant a quick pyparallel
update: a couple of weeks after PyCon, I came up with a solution for
the biggest show-stopper that has been plaguing pyparallel since its
inception: being able to detect the modification of main thread
Python objects from within a parallel context.
For example, `data.append(4)` in the example below will generate an
AssignmentError exception, because data is a main thread object, and
`data.append(4)` gets executed from within a parallel context::
data = [ 1, 2, 3 ]
def work():
data.append(4)
async.submit_work(work)
The solution turned out to be deceptively simple:
1. Prior to running parallel threads, lock all main thread
memory pages as read-only (via VirtualProtect on Windows,
mprotect on POSIX).
2. Detect attempts to write to main thread pages during parallel
thread execution (via SEH on Windows or a SIGSEGV trap on POSIX),
and raise an exception instead (detection is done in the ceval
frame exec loop).
3. Prior to returning control back to the main thread (which will
be paused whilst all the parallel threads are running), unlock
all the main thread pages.
4. Pause all parallel threads while the main thread runs.
5. Go back to 1.
I got a proof-of-concept working on Windows a while back (and also
played around with large page support in the same commit). The main
changes were to obmalloc.c:
https://bitbucket.org/tpn/pyparallel/commits/0e70a0caa1c07dc0c14bb5c99cbe808c1c11779f#chg-Objects/obmalloc.c
The key was the introduction of two new API calls, intended to be
called by the pyparallel.c infrastructure:
_PyMem_LockMainThreadPages()
_PyMem_UnlockMainThreadPages()
The implementation is pretty simple:
+int
+_PyMem_LockMainThreadPages(void)
+{
+DWORD old = 0;
+
+if (!VirtualProtect(base_addr, nbytes_committed, PAGE_READONLY, old)) {
+PyErr_SetFromWindowsErr(0);
+return -1;
+}
Note the `base_addr` and `nbytes_committed` argument. Basically, I
re-organized obmalloc.c a little bit such that we never actually
call malloc() directly. Instead, we exploit the ability to reserve
huge virtual address ranges without actually committing the memory,
giving us a fixed `base_addr` void pointer that we can pass to calls
like VirtualProtect or mprotect.
We then incrementally commit more pages as demand increases, and
simply adjust our `nbytes_committed` counter as we go along. The
net effect is that we can call VirtualProtect/mprotect once, with a
single base void pointer and size_t range, and immediately affect the
protection of all memory pages that fall within that range.
As an added bonus, we also get a very cheap and elegant way to test
if a pointer (or any arbitrary memory address, actually) belongs to
the main thread's memory range (at least in comparison to the
existing _PyMem_InRange black magic). (This is very useful for my
pyparallel infrastructure, which makes extensive use of conditional
logic based on address tests.)
(Side-bar: a side-effect of the approach I've used in the proof-
of-concept (by only having a single base addr pointer) is that
we effectively limit the maximum memory we could eventually
commit.
I actually quite like this -- in fact, I'd like to tweak it
such that we can actually expose min/max memory values to the
Python interpreter at startup (analogous to the JVM).
Having known upper bounds on maximum memory usage will vastly
simplify some other areas of my pyparallel work (like the async
socket stuff).
For example, consider network programs these days that take a
max clients configuration parameter. That seems a bit
backwards to me.
It would be better if we simply said, here, Python, you have
1GB to work with. That allows us to calculate how many
clients we could simultaneously serve based on socket memory
requirements, which allows for much more graceful behavior
under load than leaving it open-ended.
Maximum memory constraints would also be useful for the
parallel.map(callable, iterable) stuff I've got in the works,
as it'll allow us to optimally chunk work and assign to threads
based on available memory.)
So, Victor, I'm interested to hear how the new API you're proposing
will affect this solution I've come up with for pyparallel; I'm
going to be absolutely dependent upon the ability to lock main
thread pages as read-only in one fell-swoop -- am I still going to
be able to do that with your new API in place?
Regards,
Trent.
Re: [Python-Dev] pyparallel and new memory API discussions...
Hi Charles-François! Good to hear from you again. It was actually your e-mail a few months ago that acted as the initial catalyst for this memory protection idea, so, thanks for that :-) Answer below. On Wed, Jun 19, 2013 at 07:01:49AM -0700, Charles-François Natali wrote: 2013/6/19 Trent Nelson tr...@snakebite.org: The new memory API discussions (and PEP) warrant a quick pyparallel update: a couple of weeks after PyCon, I came up with a solution for the biggest show-stopper that has been plaguing pyparallel since its inception: being able to detect the modification of main thread Python objects from within a parallel context. For example, `data.append(4)` in the example below will generate an AssignmentError exception, because data is a main thread object, and `data.append(4)` gets executed from within a parallel context:: data = [ 1, 2, 3 ] def work(): data.append(4) async.submit_work(work) The solution turned out to be deceptively simple: 1. Prior to running parallel threads, lock all main thread memory pages as read-only (via VirtualProtect on Windows, mprotect on POSIX). 2. Detect attempts to write to main thread pages during parallel thread execution (via SEH on Windows or a SIGSEGV trap on POSIX), and raise an exception instead (detection is done in the ceval frame exec loop). Quick stupid question: because of refcounts, the pages will be written to even in case of read-only access. How do you deal with this? Easy: I don't refcount in parallel contexts :-) There's no need, for two reasons: 1. All memory allocated in a parallel context is localized to a private heap. When the parallel context is finished, the entire heap can be blown away in one fell-swoop. There's no need for reference counting or GC because none of the objects will exist after the parallel context completes. 2. The main thread won't be running when parallel threads/contexts are executing, which means main thread objects being accessed in parallel contexts (read-only access is fine) won't be suddenly free()'d or GC-collected or whatever. You get credit for that second point; you asked a similar question a few months ago that made me realize I absolutely couldn't have the main thread running at the same time the parallel threads were running. Once I accepted that as a design constraint, everything else came together nicely... Hmmm, if the main thread isn't running, it won't need write-access to any of its pages! If we mark them read-only, we could catch the traps/SEHs from parallel threads, then raise an exception, ahh, simple!. I'm both chuffed at how simple it is (considering it was *the* major show-stopper), and miffed at how it managed to elude me for so long ;-) Regards, Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pyparallel and new memory API discussions...
On Wed, Jun 19, 2013 at 08:45:55AM -0700, Victor Stinner wrote: 1. All memory allocated in a parallel context is localized to a private heap. How do you allocate memory in this private heap? Did you add new functions to allocate memory? Yup: _PyHeap_Malloc(): http://hg.python.org/sandbox/trent/file/0e70a0caa1c0/Python/pyparallel.c#l2365. All memory operations (PyObject_New/Malloc etc) get intercepted during parallel thread execution and redirected to _PyHeap_Malloc(), which is a very simple slab allocator. (No need for convoluted buckets because we never free individual objects during parallel execution; instead, we just blow everything away at the end.) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pyparallel and new memory API discussions...
On Wed, Jun 19, 2013 at 09:20:15AM -0700, Victor Stinner wrote:
So, Victor, I'm interested to hear how the new API you're proposing
will affect this solution I've come up with for pyparallel; I'm
going to be absolutely dependent upon the ability to lock main
thread pages as read-only in one fell-swoop -- am I still going to
be able to do that with your new API in place?
2013/6/19 Trent Nelson tr...@snakebite.org:
On Wed, Jun 19, 2013 at 08:45:55AM -0700, Victor Stinner wrote:
1. All memory allocated in a parallel context is localized to a
private heap.
How do you allocate memory in this private heap? Did you add new
functions to allocate memory?
Yup:
_PyHeap_Malloc():
http://hg.python.org/sandbox/trent/file/0e70a0caa1c0/Python/pyparallel.c#l2365.
All memory operations (PyObject_New/Malloc etc) get intercepted
during parallel thread execution and redirected to _PyHeap_Malloc(),
which is a very simple slab allocator. (No need for convoluted
buckets because we never free individual objects during parallel
execution; instead, we just blow everything away at the end.)
Ok, so I don't think that the PEP 445 would change anything for you.
The following change might have an impact: If _PyHeap_Malloc is not
thread safe, replacing PyMem_Malloc() with PyMem_RawMalloc() when the
GIL is not held would avoid bugs in your code.
Hmmm, well, _PyHeap_Malloc is sort of implicitly thread-safe, by
design, but I'm not sure if we're referring to the same sort of
thread-safe problem here.
For one, _PyHeap_Malloc won't ever run if the GIL isn't being held.
(Parallel threads are only allowed to run when the main thread has
the GIL held and has relinquished control to parallel threads.)
Also, I interpret PyMem_RawMalloc() as a direct shortcut to
malloc() (or something else that returns void *s that are then
free()'d down the track). Is that right?
I don't think that would impact pyparallel.
If you want to choose dynamically the allocator at runtime, you can
replace PyObject_Malloc allocator using:
-- 8 -
static void *
_PxMem_AllocMalloc(void *ctx, size_t size)
{
PyMemBlockAllocator *ctx;
if (Py_PXCTX)
return _PxMem_Malloc(size))
else
return alloc-malloc(alloc-ctx, size);
}
...
PyMemBlockAllocator pyparallel_pyobject;
static void *
setup_pyparallel_allocator(void)
{
PyMemBlockAllocator alloc;
PyObject_GetAllocator(pyparallel_pyobject);
alloc.ctx = pyparallel_pyobject;
alloc.malloc = _PxMem_AllocMalloc;
...
PyObject_SetAllocator(alloc);
}
-- 8 -
But I don't know if you want pyparallel to be an optional feature
chosen at runtime...
Hmmm, those code snippets are interesting. Time for some more
homework.
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slides from today's parallel/async Python talk
On Thu, Apr 04, 2013 at 11:53:01PM -0700, Charles-Fran?ois Natali wrote: Hello, async.submit_work(func, args, kwds, callback=None, errback=None) How do you implement arguments passing and return value? e.g. let's say I pass a list as argument: how do you iterate on the list from the worker thread without modifying the backing objects for refcounts (IIUC you use a per-thread heap and don't do any refcounting). Correct, nothing special is done for the arguments (apart from incref'ing them in the main thread before kicking off the parallel thread (then decref'ing them in the main thread once we're sure the parallel thread has finished)). IIUC you incref the argument from the main thread before publishing it to the worker thread: but what about containers like list? How do you make sure the refcounts of the elements don't get deallocated while the worker thread iterates? Ah, so, all of my examples were missing async.run(). They should have looked like this: async.submit_work(foo) async.submit_work(bar) async.run() async.run() is called from the main thread, with the GIL held, and it blocks until all parallel threads (well, parallel contexts, to be exact) have completed. The parallel 'work' doesn't actually start until async.run() is called either. (That's completely untrue at the moment; async.submit_work(foo) will execute foo() in a parallel thread immediately. Fixing that is on the todo list.) With only parallel threads running, no main-thread objects could ever be deallocated*, as no decref'ing is ever done. [*]: unless you went out of your way to delete/deallocate main thread objects via the @async.call_from_main_thread facility. At the moment, that's firmly in the category of Don't Do That. (And, thinking about it a little more, I guess I could augment the ceval loop in such a way that in order for the main thread to run things scheduled via @async.call_from_main_thread, all parallel threads need to be suspended. Or I could just freeze/thaw them (although I don't know if there are POSIX counterparts to those Windows methods). That would definitely impede performance, but it would assure data integrity. Perhaps it should be enabled by default, with the option to disable it for consenting adults.) More generally, how do you deal with non-local objects? Read-only ops against non-local (main-thread) objects from parallel threads are free, which is nice. Things get tricky when you try to mutate main-thread objects from parallel threads. That's where all the context persistence, interlocked data types, object protection etc stuff comes in. Is... that what you mean by how do I deal with non-local objects? I took a guess ;-) Regards, Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slides from today's parallel/async Python talk
Hi Charles-François,
On Thu, Apr 04, 2013 at 01:18:58AM -0700, Charles-François Natali wrote:
Just a quick implementation question (didn't have time to read through
all your emails :-)
async.submit_work(func, args, kwds, callback=None, errback=None)
How do you implement arguments passing and return value?
e.g. let's say I pass a list as argument: how do you iterate on the
list from the worker thread without modifying the backing objects for
refcounts (IIUC you use a per-thread heap and don't do any
refcounting).
Correct, nothing special is done for the arguments (apart from
incref'ing them in the main thread before kicking off the parallel
thread (then decref'ing them in the main thread once we're sure the
parallel thread has finished)).
Same thing for return value, how do you pass it to the
callback?
For submit_work(), you can't :-) In fact, an exception is raised if
the func() or callback() or errback() attempts to return a non-None
value.
It's worth noting that I eventually plan to have the map/reduce-type
functionality (similar to what multiprocessing offers) available via
a separate 'parallel' façade. This will be geared towards programs
that are predominantly single-threaded, but have lots of data that
can be processed in parallel at various points.
Now, with that being said, there are a few options available at the
moment if you want to communicate stuff from parallel threads back
to the main thread. Originally, you could do something like this:
d = async.dict()
def foo():
d['foo'] = async.rdtsc()
def bar():
d['bar'] = async.rdtsc()
async.submit_work(foo)
async.submit_work(bar)
But I recently identified a few memory-management flaws with that
approach (I'm still on the fence with this issue... initially I was
going to drop all support, but I've since had ideas to address the
memory issues, so, we'll see).
There's also this option:
d = dict()
@async.call_from_main_thread_and_wait
def store(k, v):
d[str(k)] = str(v)
def foo():
store('foo', async.rdtsc())
def bar():
store('bar', async.rdtsc())
async.submit_work(foo)
async.submit_work(bar)
(Not a particularly performant option though; the main-thread
instantly becomes the bottleneck.)
Post-PyCon, I've been working on providing new interlocked data
types that are specifically designed to bridge the parallel/main-
thread divide:
xl = async.xlist()
def foo():
xl.push(async.rdtsc())
def bar():
xl.push(async.rdtsc())
async.submit_work(foo)
async.submit_work(bar)
while True:
x = xl.pop()
if not x:
break
process(x)
What's interesting about xlist() is that it takes ownership of the
parallel objects being pushed onto it. That is, it basically clones
them, using memory allocated from its own internal heap (allowing
the parallel-thread's context heap to be freed, which is desirable).
The push/pop operations are interlocked at the C level, which
obviates the need for any explicit locking.
I've put that work on hold for now though; I want to finish the
async client/server stuff (it's about 60-70% done) first. Once
that's done, I'll tackle the parallel.*-type façade.
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Post-PyCon updates to PyParallel
[ python-dev: I've set up a new list for pyparallel discussions: https://lists.snakebite.net/mailman/listinfo/pyparallel. This e-mail will be the last I'll send to python-dev@ regarding the on-going pyparallel work; please drop python-dev@ from the CC and just send to pyparal...@lists.snakebite.net -- I'll stay on top of the posts-from-unsubscribed-users moderation for those that want to reply to this e-mail but not subscribe. ] Hi folks, Wanted to give a quick update on the parallel work both during and after PyCon. During the language summit when I presented the slides I uploaded to speakerdeck.com, the majority of questions from other developers revolved around the big issues like data integrity and what happens when parallel objects interact with main-thread objects and vice-versa. So, during the sprints, I explored putting guards in place to throw an exception if we detect that a user has assigned a parallel object to a non-protected main-thread object. (I describe the concept of 'protection' in my follow up posts to python-dev last week: http://mail.python.org/pipermail/python-dev/2013-March/124690.html. Basically, protecting a main-thread object allows code like this to work without crashing: d = async.dict() def foo(): # async.rdtsc() is a helper method # that basically wraps the result of # the assembly RDTSC (read time- # stamp counter) instruction into a # PyLong object. So, it's handy when # I need to test the very functionality # being demonstrated here (creating # an object within a parallel context # and persisting it elsewhere). d['foo'] = async.rdtsc() def bar(): d['bar'] = async.rdtsc() async.submit_work(foo) async.submit_work(bar) ) It was actually pretty easy, far easier than I expected. It was achieved via Px_CHECK_PROTECTION(): https://bitbucket.org/tpn/pyparallel/commits/f3fe082668c6f3f699db990f046291ff66b1b467#LInclude/object.hT1072 Various new tests related to the protection functionality: https://bitbucket.org/tpn/pyparallel/commits/f3fe082668c6f3f699db990f046291ff66b1b467#LLib/async/test/test_primitives.pyT58 The type of changes I had to make to other parts of CPython to perform the protection checks: https://bitbucket.org/tpn/pyparallel/commits/f3fe082668c6f3f699db990f046291ff66b1b467#LObjects/abstract.cT170 That was all working fine... until I started looking at adding support for lists (i.e. appending a parallel thread object to a protected, main-thread list). The problem is that appending to a list will often involve a list resize, which is done via PyMem_REALLOC() and some custom fiddling. That would mean if a parallel thread attempts to append to a list and it needs resizing, all the newly realloc'd memory would be allocated from the parallel context's heap. Now, this heap would stick around as long as the parallel objects have a refcount 0. However, as soon as the last parallel object's refcount hits 0, the entire context will be scheduled for the cleanup/release/free dance, which will eventually blow away the entire heap and all the memory allocated against that heap... which means all the **ob_item stuff that was reallocated as part of the list resize. Not particularly desirable :-) As I was playing around with ways to potentially pre-allocate lists, it occurred to me that dicts would be affected in the exact same way; I just hadn't run into it yet because my unit tests only ever assigned a few (5) objects to the protected dicts. Once the threshold gets reached (10?), a dict resize would take place, which would involve lots of PyMem_REALLOCs, and we get into the exact same situation mentioned above. So, at that point, I concluded that whole async protection stuff was not a viable long term solution. (In fact, the reason I first added it was simply to have an easy way to test things in unit tests.) The new solution I came up with: new thread-safe, interlocked data types that are *specifically* designed for this exact use case; transferring results from computation in a parallel thread back to a main thread 'container' object. First up is a new list type: xlist() (PyXListObject/PyXList_Type). I've just committed the work-in-progress stuff I've been able to hack out whilst traveling the past few days: https://bitbucket.org/tpn/pyparallel/commits/5b662eba4efe83e94d31bd9db4520a779aea612a It's not finished, and I'm pretty sure it doesn't even compile yet, but the idea is something like this:
Re: [Python-Dev] Post-PyCon updates to PyParallel
On Wed, Mar 27, 2013 at 11:26:51PM -0700, Trent Nelson wrote: [ python-dev: I've set up a new list for pyparallel discussions: https://lists.snakebite.net/mailman/listinfo/pyparallel. This e-mail will be the last I'll send to python-dev@ regarding the on-going pyparallel work; please drop python-dev@ from the CC and just send to pyparal...@lists.snakebite.net -- I'll stay on top of the posts-from-unsubscribed-users moderation for those that want to reply to this e-mail but not subscribe. ] Gah, wrong e-mail address, it's pyparal...@snakebite.net. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slides from today's parallel/async Python talk
That's good to hear :-) (It's a fantastic facility, I couldn't imagine having to go back to manual TLS API stuff after using __thread/__declspec(thread).) This e-mail was sent from a wireless device. On 21 Mar 2013, at 09:30, Baptiste Lepilleur baptiste.lepill...@gmail.commailto:baptiste.lepill...@gmail.com wrote: 2013/3/15 Trent Nelson tr...@snakebite.orgmailto:tr...@snakebite.org On Thu, Mar 14, 2013 at 03:50:27PM -0700, Martin v. Löwis wrote: Am 14.03.13 12:59, schrieb Stefan Ring: I think you should be able to just take the address of a static __thread variable to achieve the same thing in a more portable way. That assumes that the compiler supports __thread variables, which isn't that portable in the first place. FWIW, I make extensive use of __declspec(thread). I'm aware of GCC and Clang's __thread alternative. No idea what IBM xlC, Sun Studio and others offer, if anything. IBM xlC and Sun Studio also support this feature. From memory, it's also __thread keyword. This features is also supported by the new C11/C++11 standards. Baptiste. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slides from today's parallel/async Python talk
No, I haven't. I'd lose the excellent Windows pairing of thread pool IO and overlapped IO facilities if I did that. Not saying it isn't an option down the track for the generic submit work API though; that stuff will work against any thread pool without too much effort. But for now, the fact that all I need to call is TrySubmitThreadpoolCallback and Windows does *everything* else is pretty handy. Lets me concentrate on the problem instead of getting distracted by scaffolding. This e-mail was sent from a wireless device. On 21 Mar 2013, at 05:53, Sturla Molden stu...@molden.no wrote: Den 14. mars 2013 kl. 23:23 skrev Trent Nelson tr...@snakebite.org: For the record, here are all the Windows calls I'm using that have no *direct* POSIX equivalent: Interlocked singly-linked lists: - InitializeSListHead() - InterlockedFlushSList() - QueryDepthSList() - InterlockedPushEntrySList() - InterlockedPushListSList() - InterlockedPopEntrySlist() Synchronisation and concurrency primitives: - Critical sections - InitializeCriticalSectionAndSpinCount() - EnterCriticalSection() - LeaveCriticalSection() - TryEnterCriticalSection() - Slim read/writer locks (some pthread implements have rwlocks)*: - InitializeSRWLock() - AcquireSRWLockShared() - AcquireSRWLockExclusive() - ReleaseSRWLockShared() - ReleaseSRWLockExclusive() - TryAcquireSRWLockExclusive() - TryAcquireSRWLockShared() - One-time initialization: - InitOnceBeginInitialize() - InitOnceComplete() - Generic event, signalling and wait facilities: - CreateEvent() - SetEvent() - WaitForSingleObject() - WaitForMultipleObjects() - SignalObjectAndWait() Native thread pool facilities: - TrySubmitThreadpoolCallback() - StartThreadpoolIo() - CloseThreadpoolIo() - CancelThreadpoolIo() - DisassociateCurrentThreadFromCallback() - CallbackMayRunLong() - CreateThreadpoolWait() - SetThreadpoolWait() Memory management: - HeapCreate() - HeapAlloc() - HeapDestroy() Structured Exception Handling (#ifdef Py_DEBUG): - __try/__except Sockets: - ConnectEx() - AcceptEx() - WSAEventSelect(FD_ACCEPT) - DisconnectEx(TF_REUSE_SOCKET) - Overlapped WSASend() - Overlapped WSARecv() Don't get me wrong, I grew up with UNIX and love it as much as the next guy, but you can't deny the usefulness of Windows' facilities for writing high-performance, multi-threaded IO code. It's decades ahead of POSIX. (Which is also why it bugs me when I see select() being used on Windows, or IOCP being used as if it were a poll-type generic IO multiplexor -- that's like having a Ferrari and speed limiting it to 5mph!) So, before any of this has a chance of working on Linux/BSD, a lot more scaffolding will need to be written to provide the things we get for free on Windows (threadpools being the biggest freebie). Have you considered using OpenMP instead of Windows API or POSIX threads directly? OpenMP gives you a thread pool and synchronization primitives for free as well, with no special code needed for Windows or POSIX. OpenBLAS (and GotoBLAS2) uses OpenMP to produce a thread pool on POSIX systems (and actually Windows API on Windows). The OpenMP portion of the C code is wrapped so it looks like sending an asynch task to a thread pool; the C code is not littered with OpenMP pragmas. If you need something like Windows threadpools on POSIX, just look at the BSD licensed OpenBLAS code. It is written to be scalable for the world's largest supercomputers (but also beautifully written and very easy to read). Cython has code to register OpenMP threads as Python threads, in case that is needed. So that problem is also solved. Sturla ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slides from today's parallel/async Python talk
http://c2.com/cgi/wiki?BlubParadox ;-) Sent from my iPhone On 21 Mar 2013, at 06:18, Antoine Pitrou solip...@pitrou.net wrote: Le Thu, 14 Mar 2013 15:23:37 -0700, Trent Nelson tr...@snakebite.org a écrit : Don't get me wrong, I grew up with UNIX and love it as much as the next guy, but you can't deny the usefulness of Windows' facilities for writing high-performance, multi-threaded IO code. It's decades ahead of POSIX. I suppose that's why all high-performance servers run under Windows. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/trent%40snakebite.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slides from today's parallel/async Python talk
On Mon, Mar 18, 2013 at 05:27:33PM -0700, Christian Tismer wrote:
Hi Trent,
Hi Christian! Thanks for taking the time to read my walls of text
;-)
So, the remaining challenge is preventing the use case alluded to
earlier where someone tries to modify an object that hasn't been async
protected. That's a bit harder. The idea I've got in mind is to
instrument the main CPython ceval loop, such that we do these checks as
part of opcode processing. That allows us to keep all the logic in the
one spot and not have to go hacking the internals of every single
object's C backend to ensure correctness.
Now, that'll probably work to an extent. I mean, after all, there are
opcodes for all the things we'd be interested in instrumenting,
LOAD_GLOBAL, STORE_GLOBAL, SETITEM etc. What becomes challenging is
detecting arbitrary mutations via object calls, i.e. how do we know,
during the ceval loop, that foo.append(x) needs to be treated specially
if foo is a main-thread object and x is a parallel thread object?
There may be no way to handle that *other* than hacking the internals of
each object, unfortunately. So, the viability of this whole approach
may rest on whether or that's deemed as an acceptable tradeoff (a
necessary evil, even) to the Python developer community.
This is pretty much my concern:
In order to make this waterproof, as required for CPython, you will quite
likely have to do something on very many objects, and this is hard
to chime into CPython.
Actually, I think I was unnecessarily pessimistic here. When I sent
that follow-up mail with cross-references, I realized I'd forgotten
the nitty gritty details of how I implemented the async protection
support.
It turns out I'd already started on protecting lists (or rather,
PySequenceMethods), but decided to stop as the work I'd done on the
PyMappingMethods was sufficient for my needs at the time.
All I *really* want to do is raise an exception if a parallel object
gets assigned to a main-thread container object (list/dict etc) that
hasn't been async protected. (As opposed to now, where it'll
either segfault or silently corrupt stuff, then segfault later.)
I've already got all the infrastructure in place to test that (I use
it extensively within pyparallel.c):
Py_ISPY(obj) - detect a main-thread object
Py_ISPX(obj) - detect a parallel-thread object
Py_IS_PROTECTED(obj) - detect if a main-thread object has
been protected*
[*]: actually, this isn't in a macro form right now, it's a
cheeky inline:
__inline
char
_protected(PyObject *obj)
{
return (obj-px_flags Py_PXFLAGS_RWLOCK);
}
As those macros are exposed in the public Include/pyparallel.h,
they can be used in other parts of the code base. So, it's just a
matter of finding the points where an `lvalue = rvalue` takes place;
where: ``Py_ISPY(lvalue) Py_ISPX(rvalue)``. Then a test to see
if lvalue is protected; if not, raise an exception. If so, then
nothing else needs to be done.
And there aren't that many places where this happens. (It didn't
take long to get the PyMappingMethods intercepts nailed down.)
That's the idea anyway. I need to get back to coding to see how it
all plays out in practice. And there aren't many places where this
happens might be my famous last words.
If it's not, then it's unlikely this approach will ever see the light of
day in CPython. If that turns out to be the case, then I see this
project taking the path that Stackless took (forking off and becoming a
separate interpreter).
We had that discussion quite often for Stackless, and I would love to find
a solution that allows to add special versions and use cases to CPython
in a way that avoids the forking as we did it.
It would be a nice thing if we could come up with a way to keep CPython
in place, but to swap the interpreter out and replace it with a specialized
version, if the application needs it. I wonder to what extent that would be
possible.
What I would like to achieve, after having given up on Stackless integration
is a way to let it piggyback onto CPython that works like an extension
module, although it hat effectively replace larger parts of the interpreter.
I wonder if that might be the superior way to have more flexibility,
without forcing
everything and all go into CPython.
If we can make the interpreter somehow pluggable at runtime, a lot of issues
would become much simpler.
There's nothing wrong with that; I am really excited about the
possibilities afforded by this approach, and I'm sure it will pique the
interest of commercial entities out there that have problems perfectly
suited to where this pattern excels
Re: [Python-Dev] Slides from today's parallel/async Python talk
On Thu, Mar 14, 2013 at 05:21:09AM -0700, Christian Heimes wrote: Am 14.03.2013 03:05, schrieb Trent Nelson: Just posted the slides for those that didn't have the benefit of attending the language summit today: https://speakerdeck.com/trent/parallelizing-the-python-interpreter-an-alternate-approach-to-async Wow, neat! Your idea with Py_PXCTC is ingenious. Yeah, it's funny how the viability and performance of the whole approach comes down to a quirky little trick for quickly detecting if we're in a parallel thread ;-) I was very chuffed when it all fell into place. (And I hope the quirkiness of it doesn't detract from the overall approach.) As far as I remember the FS and GS segment registers are used by most modern operating systems on x86 and x86_64 platforms nowadays to distinguish threads. TLS is implemented with FS and GS registers. I guess the __read[gf]sdword() intrinsics do exactly the same. Yup, in fact, if I hadn't come up with the __read[gf]sword() trick, my only other option would have been TLS (or the GetCurrentThreadId /pthread_self() approach in the presentation). TLS is fantastic, and it's definitely an intrinsic part of the solution (the Y part of if we're a parallel thread, do Y), but it definitely more costly than a simple FS/GS register read. Reading registers is super fast and should have a negligible effect on code. Yeah the actual instruction is practically free; the main thing you pay for is the extra branch. However, most of the code looks like this: if (Py_PXCTX) something_small_and_inlineable(); else Py_INCREF(op); /* also small and inlineable */ In the majority of the cases, all the code for both branches is going to be in the same cache line, so a mispredicted branch is only going to result in a pipeline stall, which is better than a cache miss. ARM CPUs don't have segment registers because they have a simpler addressing model. The register CP15 came up after a couple of Google searches. Noted, thanks! IMHO you should target x86, x86_64, ARMv6 and ARMv7. ARMv7 is going to be more important than x86 in the future. We are going to see more ARM based servers. Yeah that's my general sentiment too. I'm definitely curious to see if other ISAs offer similar facilities (Sparc, IA64, POWER etc), but the hierarchy will be x86/x64 ARM * for the foreseeable future. Porting the Py_PXCTX part is trivial compared to the work that is going to be required to get this stuff working on POSIX where none of the sublime Windows concurrency, synchronisation and async IO primitives exist. Christian Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slides from today's parallel/async Python talk
On Wed, Mar 13, 2013 at 07:05:41PM -0700, Trent Nelson wrote: Just posted the slides for those that didn't have the benefit of attending the language summit today: https://speakerdeck.com/trent/parallelizing-the-python-interpreter-an-alternate-approach-to-async Someone on /r/python asked if I could elaborate on the do Y part of if we're in a parallel thread, do Y, if not, do X, which I (inadvertently) ended up replying to in detail. I've included the response below. (I'll work on converting this into a TL;DR set of slides soon.) Can you go into a bit of depth about X here? That's a huge topic that I'm hoping to tackle ASAP. The basic premise is that parallel 'Context' objects (well, structs) are allocated for each parallel thread callback. The context persists for the lifetime of the parallel work. The lifetime of the parallel work depends on what you're doing. For a simple ``async.submit_work(foo)``, the context is considered complete once ``foo()`` has been called (presuming no exceptions were raised). For an async client/server, the context will persist for the entirety of the connection. The context is responsible for encapsulating all resources related to the parallel thread. So, it has its own heap, and all memory allocations are taken from that heap. For any given parallel thread, only one context can be executing at a time, and this can be accessed via the ``__declspec(thread) Context *ctx`` global (which is primed by some glue code as soon as the parallel thread starts executing a callback). No reference counting or garbage collection is done during parallel thread execution. Instead, once the context is finished, it is scheduled to be released, which means it'll be processed by the main thread as part of its housekeeping work (during ``async.run()`` (technically, ``async.run_once()``). The main thread simply destroys the entire heap in one fell swoop, releasing all memory that was associated with that context. There are a few side effects to this. First, the heap allocator (basically, the thing that answers ``malloc()`` calls) is incredibly simple. It allocates LARGE_PAGE_SIZE chunks of memory at a time (2MB on x64), and simply returns pointers to that chunk for each memory request (adjusting h-next and allocation stats as it goes along, obviously). Once the 2MB has been exhausted, another 2MB is allocated. That approach is fine for the ``submit_(work|timer|wait)`` callbacks, which basically provide a way to run a presumably-finite-length function in a parallel thread (and invoking callbacks/errbacks as required). However, it breaks down when dealing with client/server stuff. Each invocation of a callback (say, ``data_received(...)``) may only consume, say, 500 bytes, but it might be called a million times before the connection is terminated. You can't have cumulative memory usage with possibly-infinite-length client/server-callbacks like you can with the once-off ``submit_(work|wait|timer)`` stuff. So, enter heap snapshots. The logic that handles all client/server connections is instrumented such that it takes a snapshot of the heap (and all associated stats) prior to invoking a Python method (via ``PyObject_Call()``, for example, i.e. the invocation of ``data_received``). When the method completes, we can simply roll back the snapshot. The heap's stats and next pointers et al all get reset back to what they were before the callback was invoked. That's how the chargen server is able to pump out endless streams of data for every client whilst keeping memory usage static. (Well, every new client currently consumes at least a minimum of 2MB (but down the track that can be tweaked back down to SMALL_PAGE_SIZE, 4096, for servers that need to handle hundreds of thousands of clients simultaneously). The only issue with this approach is detecting when the callback has done the unthinkable (from a shared-nothing perspective) and persisted some random object it created outside of the parallel context it was created in. That's actually a huge separate technical issue to tackle -- and it applies just as much to the normal ``submit_(wait|work|timer)`` callbacks as well. I've got a somewhat-temporary solution in place for that currently: d = async.dict() def foo(): # async.rdtsc() is a helper method # that basically wraps the result of # the assembly RDTSC (read time- # stamp counter) instruction into a # PyLong object. So, it's handy when # I need to test the very functionality # being demonstrated here (creating # an object within a parallel context # and persisting it elsewhere). d['foo'] = async.rdtsc() def bar(): d['bar'] = async.rdtsc() async.submit_work(foo) async.submit_work(bar) That'll result in two contexts being created, one for each callback invocation. ``async.dict()`` is a parallel safe wrapper around
Re: [Python-Dev] Slides from today's parallel/async Python talk
Cross-referenced to relevant bits of code where appropriate.
(And just a quick reminder regarding the code quality disclaimer:
I've been hacking away on this stuff relentlessly for a few months;
the aim has been to make continual forward progress without getting
bogged down in non-value-add busy work. Lots of wildly inconsistent
naming conventions and dead code that'll be cleaned up down the
track. And the relevance of any given struct will tend to be
proportional to how many unused members it has (homeless hoarder +
shopping cart analogy).)
On Thu, Mar 14, 2013 at 11:45:20AM -0700, Trent Nelson wrote:
The basic premise is that parallel 'Context' objects (well, structs)
are allocated for each parallel thread callback.
The 'Context' struct:
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel_private.h#l546
Allocated via new_context():
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l4211
also relevant, new_context_for_socket() (encapsulates a
client/server instance within a context).
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l4300
Primary role of the context is to isolate the memory management.
This is achieved via 'Heap':
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel_private.h#l281
(Which I sort of half started refactoring to use the _HEAD_EXTRA
approach when I thought I'd need to have a separate heap type for
some TLS avenue I explored -- turns out that wasn't necessary).
The context persists for the lifetime of the parallel work.
The lifetime of the parallel work depends on what you're doing. For
a simple ``async.submit_work(foo)``, the context is considered
complete once ``foo()`` has been called (presuming no exceptions were
raised).
Managing context lifetime is one of the main responsibilities of
async.run_once():
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l3841
For an async client/server, the context will persist for the entirety
of the connection.
Marking a socket context as 'finished' for servers is the job of
PxServerSocket_ClientClosed():
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l6885
The context is responsible for encapsulating all resources related to
the parallel thread. So, it has its own heap, and all memory
allocations are taken from that heap.
The heap is initialized in two steps during new_context(). First,
a handle is allocated for the underlying system heap (via
HeapCreate):
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l4224
The first heap is then initialized for use with our context via
the Heap_Init(Context *c, size_t n, int page_size) call:
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l1921
Heaps are actually linked together via a doubly-linked list. The
first heap is a value member (not a pointer) of Context; however,
the active heap is always accessed via the '*h' pointer which is
updated as necessary.
struct Heap {
Heap *prev;
Heap *next;
void *base;
void *next;
int allocated;
int remaining;
...
struct Context {
Heap heap;
Heap *h;
...
For any given parallel thread, only one context can be executing at a
time, and this can be accessed via the ``__declspec(thread) Context
*ctx`` global (which is primed by some glue code as soon as the
parallel thread starts executing a callback).
Glue entry point for all callbacks is _PyParallel_EnteredCallback:
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l3047
On the topic of callbacks, the main workhorse for the
submit_(wait|work) callbacks is _PyParallel_WorkCallback:
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l3120
The interesting logic starts at start:
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l3251
The interesting part is the error handling. If the callback raises
an exception, we check to see if an errback has been provided. If
so, we call the errback with the error details.
If the callback completes successfully (or it fails, but the errback
completes successfully), that is treated as successful callback or
errback completion, respectively:
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l3270
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l3294
If the errback fails, or no errback was provided, the exception
percolates back to the main thread. This is handled at error:
http://hg.python.org/sandbox/trent/file/7148209d5490/Python/pyparallel.c#l3300
This should make the behavior of async.run_once() clearer
Re: [Python-Dev] Slides from today's parallel/async Python talk
On Thu, Mar 14, 2013 at 02:30:14PM -0700, Trent Nelson wrote: Then it dawned on me to just add the snapshot/rollback stuff to normal Context objects. In retrospect, it's silly I didn't think of this in the first place -- the biggest advantage of the Context abstraction is that it's thread-local, but not bindingly so (as in, it'll only ever run on one thread at a time, but it doesn't matter which one, which is essential, because the ). Once I switched ... $10 if you can guess when I took a break for lunch. but it doesn't matter which one, which is essential, because there are no guarantees with regards to which thread runs which context. Is along the lines of what I was going to say. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slides from today's parallel/async Python talk
On Thu, Mar 14, 2013 at 12:59:57PM -0700, Stefan Ring wrote: Yup, in fact, if I hadn't come up with the __read[gf]sword() trick, my only other option would have been TLS (or the GetCurrentThreadId /pthread_self() approach in the presentation). TLS is fantastic, and it's definitely an intrinsic part of the solution (the Y part of if we're a parallel thread, do Y), but it definitely more costly than a simple FS/GS register read. I think you should be able to just take the address of a static __thread variable to achieve the same thing in a more portable way. Sure, but, uh, that's kinda' trivial in comparison to all the wildly unportable Windows-only functionality I'm using to achieve all of this at the moment :-) For the record, here are all the Windows calls I'm using that have no *direct* POSIX equivalent: Interlocked singly-linked lists: - InitializeSListHead() - InterlockedFlushSList() - QueryDepthSList() - InterlockedPushEntrySList() - InterlockedPushListSList() - InterlockedPopEntrySlist() Synchronisation and concurrency primitives: - Critical sections - InitializeCriticalSectionAndSpinCount() - EnterCriticalSection() - LeaveCriticalSection() - TryEnterCriticalSection() - Slim read/writer locks (some pthread implements have rwlocks)*: - InitializeSRWLock() - AcquireSRWLockShared() - AcquireSRWLockExclusive() - ReleaseSRWLockShared() - ReleaseSRWLockExclusive() - TryAcquireSRWLockExclusive() - TryAcquireSRWLockShared() - One-time initialization: - InitOnceBeginInitialize() - InitOnceComplete() - Generic event, signalling and wait facilities: - CreateEvent() - SetEvent() - WaitForSingleObject() - WaitForMultipleObjects() - SignalObjectAndWait() Native thread pool facilities: - TrySubmitThreadpoolCallback() - StartThreadpoolIo() - CloseThreadpoolIo() - CancelThreadpoolIo() - DisassociateCurrentThreadFromCallback() - CallbackMayRunLong() - CreateThreadpoolWait() - SetThreadpoolWait() Memory management: - HeapCreate() - HeapAlloc() - HeapDestroy() Structured Exception Handling (#ifdef Py_DEBUG): - __try/__except Sockets: - ConnectEx() - AcceptEx() - WSAEventSelect(FD_ACCEPT) - DisconnectEx(TF_REUSE_SOCKET) - Overlapped WSASend() - Overlapped WSARecv() Don't get me wrong, I grew up with UNIX and love it as much as the next guy, but you can't deny the usefulness of Windows' facilities for writing high-performance, multi-threaded IO code. It's decades ahead of POSIX. (Which is also why it bugs me when I see select() being used on Windows, or IOCP being used as if it were a poll-type generic IO multiplexor -- that's like having a Ferrari and speed limiting it to 5mph!) So, before any of this has a chance of working on Linux/BSD, a lot more scaffolding will need to be written to provide the things we get for free on Windows (threadpools being the biggest freebie). Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slides from today's parallel/async Python talk
On Thu, Mar 14, 2013 at 03:50:27PM -0700, Martin v. Löwis wrote: Am 14.03.13 12:59, schrieb Stefan Ring: I think you should be able to just take the address of a static __thread variable to achieve the same thing in a more portable way. That assumes that the compiler supports __thread variables, which isn't that portable in the first place. FWIW, I make extensive use of __declspec(thread). I'm aware of GCC and Clang's __thread alternative. No idea what IBM xlC, Sun Studio and others offer, if anything. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slides from today's parallel/async Python talk
On Thu, Mar 14, 2013 at 03:56:33PM -0700, Martin v. Löwis wrote:
Am 14.03.13 11:23, schrieb Trent Nelson:
Porting the Py_PXCTX part is trivial compared to the work that is
going to be required to get this stuff working on POSIX where none
of the sublime Windows concurrency, synchronisation and async IO
primitives exist.
I couldn't understand from your presentation why this is essential
to your approach. IIUC, you are just relying on the OS providing
a thread pool, (and the sublime concurrency and synchronization
routines are nothing more than that, ISTM).
Right, there's nothing Windows* does that can't be achieved on
Linux/BSD, it'll just take more scaffolding (i.e. we'll need to
manage our own thread pool at the very least).
[*]: actually, the interlocked singly-linked list stuff concerns me;
the API seems straightforward enough but the implementation becomes
deceptively complex once you factor in the ABA problem. (I'm not
aware of a portable open source alternative for that stuff.)
Implementing a thread pool on top of select/poll/kqueue seems
straight-forward.
Nod, that's exactly what I've got in mind. Spin up a bunch of
threads that sit there and call poll/kqueue in an endless loop.
That'll work just fine for Linux/BSD/OSX.
Actually, what's really interesting is the new registered IO
facilities in Windows 8/2012. The Microsoft recommendation for
achieving the ultimate performance (least amount of jitter, lowest
latency, highest throughput) is to do something like this:
while (1) {
if (!DequeueCompletionRequests(...)) {
YieldProcessor();
continue;
} else {
/* Handle requests */
}
}
That pattern looks a lot more like what you'd do on Linux/BSD (spin
up a thread per CPU and call epoll/kqueue endlessly) than any of the
previous Windows IO patterns.
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Slides from today's parallel/async Python talk
Just posted the slides for those that didn't have the benefit of attending the language summit today: https://speakerdeck.com/trent/parallelizing-the-python-interpreter-an-alternate-approach-to-async Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On Wed, Feb 27, 2013 at 08:51:16AM -0800, Michael Foord wrote: If you have other items you'd like to discuss please let me know and I can add them to the agenda. Hmm, seems like this might be a good forum to introduce the parallel/async stuff I've been working on the past few months. TL;DR version is I've come up with an alternative approach for exploiting multiple cores that doesn't rely on GIL-removal or STM (and has a negligible performance overhead when executing single-threaded code). (For those that are curious, it lives in the px branch of the sandbox/trent repo on hg.p.o, albeit in a very experimental/prototype/proof-of-concept state (i.e. it's an unorganized, undocumented, uncommented hackfest); on the plus side, it works. Sort of.) Second suggestion: perhaps a little segment on Snakebite? What it is, what's available to committers, feedback/kvetching from those who have already used it, etc. (I forgot the format of these summits -- is there a projector?) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Arena terminology (PyArena vs obmalloc.c:arena)
There's no correlation between PyArenas and the extensive use of the term arena in obmalloc.c, right? I initially assumed there was, based solely on the common use of the term arena. However, after more investigation, it *appears* as though there's absolutely no correlation. Just wanted to make 100% sure. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
[ Weird, I can't see your original e-mail Antoine; hijacking Yury's reply instead. ] On Tue, Jan 08, 2013 at 01:28:25AM -0800, Yury V. Zaytsev wrote: On Mon, 2012-05-07 at 21:49 +0200, Antoine Pitrou wrote: I guess a long time ago, threading support in operating systems wasn't very widespread, but these days all our supported platforms have it. Is it still useful for production purposes to configure --without-threads? Do people use this option for something else than curiosity of mind? All our NetBSD, OpenBSD and DragonFlyBSD slaves use --without-thread. Without it, they all wedge in some way or another. (That should be fixed*/investigated, but, until then, yeah, --without-threads allows for a slightly more useful (but still broken) test suite run on these platforms.) [*]: I suspect the problem with at least OpenBSD is that their userland pthreads implementation just doesn't cut it; there is no hope for the really technical tests that poke and prod at things like correct signal handling and whatnot. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
On Tue, Jan 08, 2013 at 06:15:45AM -0800, Stefan Krah wrote: Trent Nelson tr...@snakebite.org wrote: All our NetBSD, OpenBSD and DragonFlyBSD slaves use --without-thread. Without it, they all wedge in some way or another. (That should be fixed*/investigated, but, until then, yeah, --without-threads allows for a slightly more useful (but still broken) test suite run on these platforms.) [*]: I suspect the problem with at least OpenBSD is that their userland pthreads implementation just doesn't cut it; there is no hope for the really technical tests that poke and prod at things like correct signal handling and whatnot. For OpenBSD the situation should be fixed in the latest release: http://www.openbsd.org/52.html#new I haven't tried it myself though. Interesting! I'll look into upgrading the existing Snakebite OpenBSD slaves (they're both at 5.1). Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PandaBoard, Raspberry Pi coming to Buildbot fleet
On Thu, Dec 20, 2012 at 09:33:27AM -0800, Brian Curtin wrote: Last week in Raymond's dictionary thread, the topic of ARM came up, along with the relative lack of build slave coverage. Today Trent Nelson received the PandaBoard purchased by the PSF, and a Raspberry Pi should be coming shortly as well. http://blog.python.org/2012/12/pandaboard-raspberry-pi-coming-to.html Thanks to the PSF for purchasing and thanks to Trent for offering to host them in Snakebite! The installation of Ubuntu on the Pandaboard went smoothly. However, it crashes after about an hour. Console output: http://trent.snakebite.net/pandaboard-crash.txt Any ARM wizards out there with suggestions? Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PandaBoard, Raspberry Pi coming to Buildbot fleet
On Fri, Jan 04, 2013 at 02:48:00AM -0800, Trent Nelson wrote: On Thu, Dec 20, 2012 at 09:33:27AM -0800, Brian Curtin wrote: Last week in Raymond's dictionary thread, the topic of ARM came up, along with the relative lack of build slave coverage. Today Trent Nelson received the PandaBoard purchased by the PSF, and a Raspberry Pi should be coming shortly as well. http://blog.python.org/2012/12/pandaboard-raspberry-pi-coming-to.html Thanks to the PSF for purchasing and thanks to Trent for offering to host them in Snakebite! The installation of Ubuntu on the Pandaboard went smoothly. However, it crashes after about an hour. Console output: http://trent.snakebite.net/pandaboard-crash.txt Any ARM wizards out there with suggestions? Forgot to mention, it's accessible via the p1 alias from the Snakebite menu. Until it crashes, anyway :-) If anyone wants root access to poke around, let me know. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PandaBoard, Raspberry Pi coming to Buildbot fleet
On Fri, Jan 04, 2013 at 05:06:22AM -0800, Victor Stinner wrote: 2013/1/4 Trent Nelson tr...@snakebite.org: The installation of Ubuntu on the Pandaboard went smoothly. However, it crashes after about an hour. Console output: http://trent.snakebite.net/pandaboard-crash.txt Any ARM wizards out there with suggestions? The bug was already reported to Ubuntu: https://bugs.launchpad.net/ubuntu/+source/linux-meta-ti-omap4/+bug/1012735 Ah! Thanks. The issue contains a workaround. [root@manganese/ttypts/0(~)#] echo performance /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor [root@manganese/ttypts/0(~)#] echo performance /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor Doesn't get more Linuxy than that :P Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible GIL/threading issue involving subprocess and PyMem_MALLOC...
On Fri, Dec 21, 2012 at 01:43:11AM -0800, Antoine Pitrou wrote: Le Fri, 21 Dec 2012 09:31:44 +, Kristján Valur Jónsson krist...@ccpgames.com a écrit : I ran into this the other day. I had put in hooks in the PyMem_MALLOC to track memory per tasklet, and it crashed in those cases because it was being called without the GIL. My local patch was simply to _not_ release the GIL. Clearly, calling PyMem_MALLOC without the GIL is an API violation. Indeed, this deserves fixing. (it would be better to still release the GIL around the low-level I/O call, of course) Created http://bugs.python.org/issue16742 to capture the issue for now. I want to make some more progress on the parallel stuff first so if somebody wants to tackle it in the meantime, be my guest. Thanks Trent for finding this! Unexpected (but handy) side-effect of the parallel context work :-) (I wonder if that's the only thread-safe issue in our code base...) Antoine. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Possible GIL/threading issue involving subprocess and PyMem_MALLOC...
This seems odd to me so I wanted to see what others think. The unit
test Lib/unittest/test/test_runner.py:Test_TextRunner.test_warnings
will eventually hit subprocess.Popen._communicate.
The `mswindows` implementation of this method relies on threads to
buffer stdin/stdout. That'll eventually result in PyOs_StdioReadline
being called without the GIL being held. PyOs_StdioReadline calls
PyMem_MALLOC, PyMem_FREE and possibly PyMem_REALLOC.
On a debug build, these macros are redirected to their _PyMem_Debug*
counterparts. The call hierarchy for _PyMem_DebugMalloc looks like
this:
void *
_PyMem_DebugMalloc(size_t nbytes)
{
return _PyObject_DebugMallocApi(_PYMALLOC_MEM_ID, nbytes);
}
/* generic debug memory api, with an id to
identify the API in use */
void *
_PyObject_DebugMallocApi(char id, size_t nbytes)
{
uchar *p; /* base address of malloc'ed block */
uchar *tail;/* p + 2*SST + nbytes ==
pointer to tail pad bytes */
size_t total; /* nbytes + 4*SST */
bumpserialno();
^^^
total = nbytes + 4*SST;
if (total nbytes)
/* overflow: can't represent total as a size_t */
return NULL;
p = (uchar *)PyObject_Malloc(total);
-^^^
if (p == NULL)
return NULL;
snip
Both bumpserialno() and PyObject_Malloc affect global state. The latter
also has a bunch of LOCK() and UNLOCK() statements, but these end up being
no-ops:
/*
* Python's threads are serialized,
* so object malloc locking is disabled.
*/
#define SIMPLELOCK_DECL(lock) /* simple lock declaration */
#define SIMPLELOCK_INIT(lock) /* allocate (if needed) and ... */
#define SIMPLELOCK_FINI(lock) /* free/destroy an existing */
#define SIMPLELOCK_LOCK(lock) /* acquire released lock */
#define SIMPLELOCK_UNLOCK(lock) /* release acquired lock */
...
/*
* This malloc lock
*/
SIMPLELOCK_DECL(_malloc_lock)
#define LOCK() SIMPLELOCK_LOCK(_malloc_lock)
#define UNLOCK()SIMPLELOCK_UNLOCK(_malloc_lock)
#define LOCK_INIT() SIMPLELOCK_INIT(_malloc_lock)
#define LOCK_FINI() SIMPLELOCK_FINI(_malloc_lock)
The PyObject_Malloc() one concerns me the most, as it affects huge
amounts of global state. Also, I just noticed PyOs_StdioReadline()
can call PyErr_SetString, which will result in a bunch of other
calls that should only be made whilst the GIL is held.
So, like I said, this seems like a bit of a head scratcher. Legit
issue or am I missing something?
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PandaBoard, Raspberry Pi coming to Buildbot fleet
On Thu, Dec 20, 2012 at 10:10:45AM -0800, Benjamin Peterson wrote: 2012/12/20 Eli Bendersky eli...@gmail.com: On Thu, Dec 20, 2012 at 9:33 AM, Brian Curtin br...@python.org wrote: Last week in Raymond's dictionary thread, the topic of ARM came up, along with the relative lack of build slave coverage. Today Trent Nelson received the PandaBoard purchased by the PSF, and a Raspberry Pi should be coming shortly as well. http://blog.python.org/2012/12/pandaboard-raspberry-pi-coming-to.html Thanks to the PSF for purchasing and thanks to Trent for offering to host them in Snakebite! __ That's good news. A related question about Snakebite, though. Maybe I missed something obvious, but is there an overview of how the core devs can use it? In particular, I'd want to know if Snakebite runs Python's tests regularly - and if it does, how can I see the status. How do I know if any commit of mine broke some host Snakebite has? How can I SSH to that host in order to reproduce and fix the problem? Some sort of a blog post about this, at least, would be very helpful for me and possibly other developers as well. http://mail.python.org/pipermail/python-dev/2012-September/121651.html Presumably that should go somewhere more permanent. Indeed, I'm going to carve out some time over the Christmas/NY break to work on this. There should really be a Developer's Guide that explains how to get the most out of the network. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PandaBoard, Raspberry Pi coming to Buildbot fleet
On Thu, Dec 20, 2012 at 09:33:27AM -0800, Brian Curtin wrote: Last week in Raymond's dictionary thread, the topic of ARM came up, along with the relative lack of build slave coverage. Today Trent Nelson received the PandaBoard purchased by the PSF, and a Raspberry Pi should be coming shortly as well. http://blog.python.org/2012/12/pandaboard-raspberry-pi-coming-to.html Thanks to the PSF for purchasing and thanks to Trent for offering to host them in Snakebite! No problemo'. If only all the other Snakebite servers could fit in my palm and run off 0.25A. (The HP-UX and Tru64 boxes in particular take up 7U/8U, weigh 160/140lbs, and chew about ~12A each.) I'll work on setting the ARM boards up next week. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PandaBoard, Raspberry Pi coming to Buildbot fleet
On Thu, Dec 20, 2012 at 10:52:56AM -0800, Trent Nelson wrote: I'll work on setting the ARM boards up next week. Does anyone have a preference regarding the operating system? There are a bunch of choices listed here: http://www.omappedia.org/wiki/Main_Page As long as it can run a recent sshd and zsh, I have no preference. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible GIL/threading issue involving subprocess and PyMem_MALLOC...
On Thu, Dec 20, 2012 at 05:47:40PM -0800, Gregory P. Smith wrote: On Thu, Dec 20, 2012 at 10:43 AM, Trent Nelson tr...@snakebite.org wrote: This seems odd to me so I wanted to see what others think. The unit test Lib/unittest/test/test_runner.py:Test_TextRunner.test_warnings will eventually hit subprocess.Popen._communicate. The `mswindows` implementation of this method relies on threads to buffer stdin/stdout. That'll eventually result in PyOs_StdioReadline being called without the GIL being held. PyOs_StdioReadline calls PyMem_MALLOC, PyMem_FREE and possibly PyMem_REALLOC. Those threads are implemented in Python so how would the GIL ever not be held? -gps PyOS_Readline drops the GIL prior to calling PyOS_StdioReadline: Py_BEGIN_ALLOW_THREADS ^^ #ifdef WITH_THREAD PyThread_acquire_lock(_PyOS_ReadlineLock, 1); #endif /* This is needed to handle the unlikely case that the * interpreter is in interactive mode *and* stdin/out are not * a tty. This can happen, for example if python is run like * this: python -i test1.py */ if (!isatty (fileno (sys_stdin)) || !isatty (fileno (sys_stdout))) rv = PyOS_StdioReadline (sys_stdin, sys_stdout, prompt); -^^^ else rv = (*PyOS_ReadlineFunctionPointer)(sys_stdin, sys_stdout, prompt); Py_END_ALLOW_THREADS Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PandaBoard, Raspberry Pi coming to Buildbot fleet
On Thu, Dec 20, 2012 at 03:10:49PM -0800, Eli Bendersky wrote: That's good news. A related question about Snakebite, though. Maybe I missed something obvious, but is there an overview of how the core devs can use it? In particular, I'd want to know if Snakebite runs Python's tests regularly - and if it does, how can I see the status. How do I know if any commit of mine broke some host Snakebite has? How can I SSH to that host in order to reproduce and fix the problem? Some sort of a blog post about this, at least, would be very helpful for me and possibly other developers as well. http://mail.python.org/pipermail/python-dev/2012-September/121651.html Presumably that should go somewhere more permanent. Indeed, I'm going to carve out some time over the Christmas/NY break to work on this. There should really be a Developer's Guide that explains how to get the most out of the network. Thanks, indeed a more permanent place would be nice. So from reading the above, am I correct in the understanding that these hosts don't actually run tests at the moment? They only do if we log into them to test stuff? I think it would be really nice if they could actually run as buildbot slaves and execute Python tests continuously. Almost all of them are running slaves, and have been since ~August. Take a look at our buildbot page -- any host with [SB] in the name is a Snakebite host. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Mercurial workflow question...
Scenario: I'm working on a change that I want to actively test on a bunch of Snakebite hosts. Getting the change working is going to be an iterative process -- lots of small commits trying to attack the problem one little bit at a time. Eventually I'll get to a point where I'm happy with the change. So, it's time to do all the necessary cruft that needs to be done before making the change public. Updating docs, tweaking style, Misc/NEWS, etc. That'll involve at least a few more commits. Most changes will also need to be merged to other branches, too, so that needs to be taken care of. (And it's a given that I would have been pulling and merging from hg.p.o/cpython during the whole process.) Then, finally, it's time to push. Now, if I understand how Mercurial works correctly, using the above workflow will result in all those little intermediate hacky commits being forever preserved in the global/public cpython repo. I will have polluted the history of all affected files with all my changes. That just doesn't feel right. But, it appears as though it's an intrinsic side-effect of how Mercurial works. With git, you have a bit more flexibility to affect how your final public commits via merge fast-forwarding. Subversion gives you the ultimate control of how your final commit looks (albeit at the expense of having to do the merging in a much more manual fashion). As I understand it, even if I contain all my intermediate commits in a server-side cloned repo, that doesn't really change anything; all commits will eventually be reflected in cpython via the final `hg push`. So, my first question is this: is this actually a problem? Is the value I'm placing on pristine log histories misplaced in the DVCS world? Do we, as a collective, care? I can think of two alternate approaches I could use: - Use a common NFS mount for each source tree on every Snakebite box (and coercing each build to be done in a separate area). Get everything perfect and then do a single commit of all changes. The thing I don't like about this approach is that I can't commit/rollback/tweak/bisect intermediate commits as I go along -- some changes are complex and take a few attempts to get right. - Use a completely separate clone to house all the intermediate commits, then generate a diff once the final commit is ready, then apply that diff to the main cpython repo, then push that. This approach is fine, but it seems counter-intuitive to the whole concept of DVCS. Thoughts? Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] More compact dictionaries with faster iteration
On Mon, Dec 10, 2012 at 08:53:17AM -0800, Barry Warsaw wrote: On Dec 10, 2012, at 05:39 PM, Christian Heimes wrote: I had ARM platforms in mind, too. Unfortunately we don't have any ARM platforms for testing and performance measurement. Even Trent's Snakebite doesn't have ARM boxes. I've a first generation Raspberry Pi, that's all. Perhaps somebody (PSF ?) is willing to donate a couple of ARM boards to Snakebite. I'm thinking of Raspberry Pi (ARMv6), Pandaboard (ARMv7 Cortex-A9) and similar. Suitable ARM boards can be had cheap, probably overwhelmed by labor costs of getting them up and running. I am not offering for *that*. ;) If someone donates the hardware, I'll take care of everything else. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python 3.3 module test failures on FreeBSD 9 amd64
On Sat, Dec 08, 2012 at 01:07:43PM -0800, A G wrote: Ideally I'd like to resolve these and submit a port for python3.3 since the most recent FreeBSD port is stuck on 3.2. FWIW, the FreeBSD Python port maintainer, Kubilay Kocak, is active on #python-dev@freenode under the nick 'koobs'. He has been working on the 3.3 port. I'd recommend liaising with him in order to avoid duplicating any effort. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tru64 support
On Tue, Sep 04, 2012 at 02:24:10AM -0700, Andrew Svetlov wrote: Is it still up to date? Bug has been created in 2004. I don't see Tru64 in list of available buildbots. Do we need to care about this platform? And how to make sure what existing code works fine for that? There's a Snakebite Tru64 box now, accessible via t5 from the sb menu. 2.7 compiles fine but hangs on some of the subprocess epipe tests, I'll set up a slave once I've addressed that. One nice thing about Tru64 is that everything is implicitly 64-bit. There's no notion of dual 32-bit/64-bit libs/binaries or a 32-bit memory architecture. The 64-bit `python` on Tru64 is in a much better state than said binaries on AIX and HP-UX (which still need a lot of work). As for VMS/OpenVMS, still working with HP to get access to media. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Handling support for newer OS features at run time
The hackiest part of my WSAPoll patch is this: --- a/PC/pyconfig.h Sun Nov 18 10:42:42 2012 + +++ b/PC/pyconfig.h Sun Nov 18 17:27:29 2012 -0500 @@ -158,12 +158,12 @@ /* set the version macros for the windows headers */ #ifdef MS_WINX64 /* 64 bit only runs on XP or greater */ -#define Py_WINVER 0x0501 /* _WIN32_WINNT_WINXP */ -#define Py_NTDDI NTDDI_WINXP +#define Py_WINVER 0x0600 /* _WIN32_WINNT_WINXP */ +#define Py_NTDDI NTDDI_WIN6 #else /* Python 2.6+ requires Windows 2000 or greater */ -#define Py_WINVER 0x0500 /* _WIN32_WINNT_WIN2K */ -#define Py_NTDDI NTDDI_WIN2KSP4 +#define Py_WINVER 0x0600 /* _WIN32_WINNT_WIN2K */ +#define Py_NTDDI NTDDI_WIN6 #endif Basically, because of the way our build is currently configured, I had to bump the minimum supported Windows version from XP to Vista just to get access to the WSAPoll headers. (Issue: http://bugs.python.org/issue16507 Patch: http://bugs.python.org/file28038/wsapoll.patch) Ideally, a Windows binary should make WSAPoll/select.poll() available if running on Vista or above, without impacting the ability to run on XP. I don't think we've currently got the ability to do this, right? Is there a precedent set anywhere else? I suspect it's not as much of an issue on *NIX platforms as you'll typically compile from source. Windows, not so much. Thoughts? A single binary that dynamically loads applicable modules seems cleaner but will obviously take more work. Other approach would be to start distributing multiple versions of Python based on the underlying Windows version. Not the nicest approach. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Handling support for newer OS features at run time
On Tue, Nov 27, 2012 at 03:09:12PM -0800, Matthias Klose wrote: Am 27.11.2012 23:49, schrieb Trent Nelson: I don't think we've currently got the ability to do this, right? Is there a precedent set anywhere else? I suspect it's not as much of an issue on *NIX platforms as you'll typically compile from source. Windows, not so much. Thoughts? A single binary that dynamically loads applicable modules seems cleaner but will obviously take more work. Other approach would be to start distributing multiple versions of Python based on the underlying Windows version. Not the nicest approach. Usually I have to build a python package on a build daemon running the kernel of the latest stable (or latest stable long term support) release. Thus if a configure test checks the current kernel, then you may get an unexpected answer and a missing feature. It is somehow different that I already build different binaries (per release), but the hard-coding of kernel features during configure time looks like the same issue. Currently working around it by patching configure to remove the test and hard-code it for a particular build. Another solution maybe would to have something like --enable-kernel=version (as found in glibc), and hope that you can deduce the features from the kernel version. Hmmm. How often do Linux kernel versions expose new features that we can make use of in Python? i.e. do you run into this problem often? Can you cite some recent examples? I'm not sure how much could be shared between Windows and Linux with what you've just described. With Windows, specifically with regards to providing dynamic select.poll() support, I was thinking of having multiple modules: Python33\ DLLs\ select.pyd select_win6.pyd select_win7.pyd select_win8.pyd And Python would automatically import the appropriate one. I don't think this would be that useful on Linux -- as you say, the decision is typically made at ./configure time. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Handling support for newer OS features at run time
On Tue, Nov 27, 2012 at 03:14:00PM -0800, Richard Oudkerk wrote: On 27/11/2012 10:49pm, Trent Nelson wrote: Ideally, a Windows binary should make WSAPoll/select.poll() available if running on Vista or above, without impacting the ability to run on XP. I assume you can do something like int WSAAPI (*pWSAPoll)(WSAPOLLFD *, ULONG, INT); HINSTANCE hWinsock2 = GetModuleHandle(WS2_32); *(FARPROC *)pWSAPoll = GetProcAddress(hWinsock2, WSAPoll); if (pWSAPoll == NULL) ... to get a function pointer for WSAPoll on versions which support it. Ah! Good point, that's probably a much better approach in this instance, given I only need one or two more symbols. Thanks! Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] externals?
On Sun, Nov 18, 2012 at 04:18:43AM -0800, mar...@v.loewis.de wrote: I'd like to stress that we don't need any versioning here. wget and tar would be sufficient, except that it's Windows, so we have neither wget nor tar. However, including a PowerShell script may be an option; most developers will have PowerShell already on their system. AFAICT, PowerShell can do HTTP downloads and extract zip files. I'm disturbed that I subconsciously interpreted this as a challenge to do it via a standalone batch/.bat ;-) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] externals?
On Thu, Nov 15, 2012 at 01:20:09AM -0800, Kristj?n Valur J?nsson wrote: Perhaps the unix makefiles get the proper version, but a windows developer has to fetch those externals manually. Pro-tip: if you're developing on Windows, you're mad if you don't prime your dev env with Tools\buildbot\externals.bat. It takes care of *everything*. I wish every proprietary UNIX system we support had something similar. Also, is there any reason to keep this in svn? I think it's more a case of there being no tangible benefit (and numerous drawbacks) to switching it to hg. I personally have no need for a local hg repo with 30 different Tcl/Tk histories in it. Subversion's good for this sort of use case. The externals repo gets committed to maybe, what, 3-4 times a year? Why not check this in to HG, we need not worry about history, etc. Who are these mystical people worried about history? ;-) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ctypes is not an acceptable implementation strategy for modules in the standard library?
On Mon, Nov 05, 2012 at 03:36:00AM -0800, Victor Stinner wrote: I'm not sure that ctypes is always available (available on all platforms). Indeed. Every non-x86 Snakebite platform has pretty serious issues with ctypes. I spent a morning looking into one platform, Solaris 10 on SPARC, and, after building the latest libffi from scratch, it failed more tests than that box chews amps. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bumping autoconf from 2.68 to 2.69
On Tue, Oct 16, 2012 at 12:12:35AM -0700, R. David Murray wrote: On Mon, 15 Oct 2012 23:18:04 -0700, Ned Deily n...@acm.org wrote: In article 20121016043352.ga21...@snakebite.org, Trent Nelson tr...@snakebite.org wrote: Any objections to regenerating configure with autoconf 2.69? The current version is based off 2.68, which was release on the 22nd of September 2010. 2.69 was released on the 24th of April, 2012. (There are some fixes for the more esoteric UNIX platforms that Snakebite will benefit from. Also, I struggled to find a box with 2.68 even installed when I needed to regenerate configure this weekend; it seems all package management/ports have bumped to 2.69.) If there are no objections, can it be applied across the board? 2.7, 3.2, 3.3 and 3.x? I believe it is the case that we don't currently require a specific version of autoconf. autoconf is only run when making a committer makes a change to configure.ac et al and then it is run on the committer's machine. So whatever is checked in for those files reflects the version of autoconf that the committer used. My understanding is that we use a specific version of autoconf. The reason is that otherwise we end up with useless churn in the repo as the generated file changes when different committers use different versions. Indeed; the 2.68-2.69 bump results in a 746 line diff for configure. In the past we have had issues with a new autoconf version actually breaking the Python build, so we also need to test a new version before switching to it. If there are no objections in principle to the bump (i.e. nobody is aware of any existing 2.69 incompatibilities), my plan is to bump, then make sure all the build slaves and Snakebite machines work with the new version. If 2.69 introduces insurmountable build issues, I'll simply revert back to 2.68 (with some notes at the top of configure.ac mentioning the issues with 2.69). Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bumping autoconf from 2.68 to 2.69
On Tue, Oct 16, 2012 at 01:04:46AM -0700, Ned Deily wrote: In article 20121016071236.0792d250...@webabinitio.net, R. David Murray rdmur...@bitdance.com wrote: My understanding is that we use a specific version of autoconf. The reason is that otherwise we end up with useless churn in the repo as the generated file changes when different committers use different versions. In the past we have had issues with a new autoconf version actually breaking the Python build, so we also need to test a new version before switching to it. Well, there was a test in the past for a specific version of autoconf but I believe that was removed. I see that the default branch configure.ac contains AC_PREREQ(2.65) but the most recently checked-in version of configure was generated with 2.68. That's a good point regarding AC_PREREQ. I think there's been a general unspoken rule to try and minimize 'configure' churn, which means using the same version of autoconf that the previous person used. I understand about the desirability of avoiding a lot of churn although it seems to me to not be a huge thing; I don't know, the churn is usually offensive enough to warrant using the same autoconf version. It's impossible to vet a configure.ac change if you're regenerating configure with a different version of autoconf. build breaking is another matter, of course. If we are going to mandate a specific version again, that should be documented and checked for. My preference: bump to 2.69 and set AC_PREREQ(2.69). If 2.69 proves unworkable, revert back to 2.68 and AC_PREREQ(2.68). I definitely like the idea of explicitly setting the version being used via AC_PREREQ, as this will prevent accidental version churn being checked in by a committer inadvertently. -- Ned Deily, n...@acm.org Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bumping autoconf from 2.68 to 2.69
On Tue, Oct 16, 2012 at 01:43:37AM -0700, Charles-François Natali wrote: My understanding is that we use a specific version of autoconf. The reason is that otherwise we end up with useless churn in the repo as the generated file changes when different committers use different versions. In the past we have had issues with a new autoconf version actually breaking the Python build, so we also need to test a new version before switching to it. Well, so I guess all committers will have to use the same Linux/FreeBSD/whatever distribution then? AFAICT there's no requirement regarding the mercurial version used by committers either. Autoconf is a special case though. Different versions of autoconf produce wildly different outputs for 'configure', making it impossible to vet configure.ac changes by reviewing the configure diff. It doesn't matter which OS or Mercurial version a developer uses as they don't implicitly affect any versioned resources; autoconf does. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bumping autoconf from 2.68 to 2.69
On Tue, Oct 16, 2012 at 12:12:35AM -0700, R. David Murray wrote: On Mon, 15 Oct 2012 23:18:04 -0700, Ned Deily n...@acm.org wrote: In article 20121016043352.ga21...@snakebite.org, Trent Nelson tr...@snakebite.org wrote: Any objections to regenerating configure with autoconf 2.69? The current version is based off 2.68, which was release on the 22nd of September 2010. 2.69 was released on the 24th of April, 2012. (There are some fixes for the more esoteric UNIX platforms that Snakebite will benefit from. Also, I struggled to find a box with 2.68 even installed when I needed to regenerate configure this weekend; it seems all package management/ports have bumped to 2.69.) If there are no objections, can it be applied across the board? 2.7, 3.2, 3.3 and 3.x? I believe it is the case that we don't currently require a specific version of autoconf. autoconf is only run when making a committer makes a change to configure.ac et al and then it is run on the committer's machine. So whatever is checked in for those files reflects the version of autoconf that the committer used. My understanding is that we use a specific version of autoconf. The reason is that otherwise we end up with useless churn in the repo as the generated file changes when different committers use different versions. In the past we have had issues with a new autoconf version actually breaking the Python build, so we also need to test a new version before switching to it. Just to put this in perspective, here's a change I made to configure.ac over the weekend as part of #15819: % hg diff configure.ac diff -r 51ce9830d85a configure.ac --- a/configure.ac Sat Oct 13 11:58:23 2012 -0400 +++ b/configure.ac Tue Oct 16 09:12:56 2012 + @@ -9,6 +9,9 @@ AC_INIT(python, PYTHON_VERSION, http://bugs.python.org/) +BUILDDIR=`pwd` +AC_SUBST(BUILDDIR) + AC_SUBST(HGVERSION) AC_SUBST(HGTAG) AC_SUBST(HGBRANCH) With the correct version of autoconf (2.68), this generates the following diff to configure: % hg diff configure diff -r 51ce9830d85a configure --- a/configure Sat Oct 13 11:58:23 2012 -0400 +++ b/configure Tue Oct 16 09:15:04 2012 + @@ -714,6 +714,7 @@ HGBRANCH HGTAG HGVERSION +BUILDDIR target_alias host_alias build_alias @@ -2719,6 +2720,9 @@ +BUILDDIR=`pwd` + + However, the 2.69 version of autoconf generates this monster of a configure diff: http://hydrogen.snakebite.net/~cpython/configure.diff As you can see, it's impossible to figure out what my configure.ac change was from that diff. Which is why I'm proposing we leverage AC_PREREQ to prevent inadvertent autoconf version churn. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bumping autoconf from 2.68 to 2.69
On Tue, Oct 16, 2012 at 02:17:35AM -0700, Charles-François Natali wrote: It should be sufficient to install autoconf-x.y into /home/user/bin or something similar. Installing autoconf from source really takes about 3 minutes. Well, maybe, maybe not. autoconf depends on a least m4 and Perl, and you may very well have a compatibility issue here. That's why most distributions have package managers, and in 2012 we're past the './configure make make install. If you're unable to get 2.69 (or whatever) installed for whatever reason, you can always just commit configure.ac and ask another committer to regenerate configure with the correct version. (In my case, I couldn't find a single Snakebite box that had 2.68 installed; out of 20-something hosts, everything had 2.69, which is partly why I'd like to bump to 2.69.) It doesn't matter which OS or Mercurial version a developer uses as they don't implicitly affect any versioned resources; autoconf does. If you're worried about the noise in diff, it's never been a problem at least to me (just don't post a configure diff for review, the configure.ac is enough). If you're worried about runtime compatibility, then autoconf is not your only worry. Proper build also depends on the target shell, target toolchain (gcc, libc, etc). Ignoring diff noise, what I'm suggesting is that we formalize the version used to build configure, as opposed to just accepting that configure was built with whatever arbitrary version of autoconf the last developer that modified configure.ac had installed. If you look at configure's history, we pretty much already follow this informally. Anyway, back to the original question: does anyone know of reasons we shouldn't bump to 2.69? Any known incompatibilities? Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bumping autoconf from 2.68 to 2.69
On Tue, Oct 16, 2012 at 08:23:00AM -0700, Brett Cannon wrote: On Tue, Oct 16, 2012 at 11:18 AM, Tres Seaver tsea...@palladion.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/16/2012 09:47 AM, Barry Warsaw wrote: On Oct 16, 2012, at 05:32 AM, Trent Nelson wrote: Anyway, back to the original question: does anyone know of reasons we shouldn't bump to 2.69? A Any known incompatibilities? There will be problems building with 2.69 on Ubuntus older than 12.10, and Debians older than wheezy. % rmadison autoconf autoconf | A A 2.61-4 | A A A A hardy | source, all autoconf | 2.65-3ubuntu1 | A A A A lucid | source, all autoconf | 2.67-2ubuntu1 | A A A A natty | source, all autoconf | 2.68-1ubuntu1 | oneiric | source, all autoconf | 2.68-1ubuntu2 | A A A precise | source, all autoconf | 2.69-1ubuntu1 | A A A quantal | source, all % rmadison -u debian autoconf autoconf | 2.67-2 | squeeze | source, all autoconf | 2.69-1 | wheezy A | source, all autoconf | 2.69-1 | sid | source, all FWIW, precise is Ubuntu 12.04 LTS, so it carries Python 2.7 and 3.2. I think it would be fine to update the default branch (i.e. 3.4), but I'm not sure what benefit you gain from making this change to stable branches, and you could potentially cause build problems, which you may not find out about for a while, e.g. when 2.7.4 is released and all the distros go to update. Agreed: A this is really the same issue as bumping the VisualStudio version (or any other build tooling) inside a release line: A too much potential for breakage for little gain. I think Barry's suggestion of updating default and leaving stable versions alone is a good one.A Bumped 3.4 to 2.69 in 79763:74b95194ba86! Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Bumping autoconf from 2.68 to 2.69
Any objections to regenerating configure with autoconf 2.69? The current version is based off 2.68, which was release on the 22nd of September 2010. 2.69 was released on the 24th of April, 2012. (There are some fixes for the more esoteric UNIX platforms that Snakebite will benefit from. Also, I struggled to find a box with 2.68 even installed when I needed to regenerate configure this weekend; it seems all package management/ports have bumped to 2.69.) If there are no objections, can it be applied across the board? 2.7, 3.2, 3.3 and 3.x? Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] buildbot with -O
Just as an FYI, we set up a new optimized Snakebite slave yesterday on the Mac OS X Mountain Lion box. I also set up remote access to that box in the process -- you can connect to it via the ~/.snakebite/sb 'x8' alias. (That box happens to be the fastest (and most contemporary) Snakebite machine; Mac Pro circa late 2009, dual quad core 3.2GHz Xenons (12MB L2 cache on each CPU!), 32GB RAM. See, it's not just a network of old esoteric junk, honest ;-).) Regards, Trent. On Wed, Sep 26, 2012 at 04:00:53PM -0700, Benjamin Peterson wrote: In light of issue #16046, I think it would be helpful to have a buildbot running the testsuite with -O enabled. -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/trent%40snakebite.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] buildbot with -O
On Wed, Sep 26, 2012 at 04:13:42PM -0700, Benjamin Peterson wrote: 2012/9/26 Maciej Fijalkowski fij...@gmail.com: On Thu, Sep 27, 2012 at 1:00 AM, Benjamin Peterson benja...@python.org wrote: In light of issue #16046, I think it would be helpful to have a buildbot running the testsuite with -O enabled. How about deprecating -O completely instead? It does not seem to give any speedups and definitely gives headaches. I'm all for that, but I'd rather have it tested in the meantime. You can use Snakebite for this sort of stuff. Grab a slave ID off Antoine, then log in to whatever slave you want and set up a new instance (say, ~/buildslave-with-O). (Look at existing slave details in ~/buildslave/buildbot.tac for hints on password etc.) And as a general side bar: all committers automatically have full access to Snakebite. You don't need to register keys* or sign up or anything. I've tried to make it as painless as possible to connect to: % cd ~ % svn co http://svn.snakebite.net/.snakebite % cd .snakebite % sh snakebite.subr [*]: just to clarify: Snakebite's already been primed with whatever ssh key you use to commit to hg. From then on, just run ~/.snakebite/sb to get access to the host menu. That will work out of the box for all committers. (Next set of enhancements will focus on linking a local source directory (on your laptop/desktop) to Snakebite and spinning up a buildbot master + slaves on demand, so you won't even have to log in and faff around with priming a build environment like you do now. (I, uh, sort of need to do a little funding/donation drive before then though, which is what I'm currently organizing.)) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Snakebite v0.1: ready for beta testing.
Quick start: % cd ~ svn co http://svn.snakebite.net/.snakebite cd .snakebite sh snakebite.subr If all goes well, you should see something like this: A.snakebite/snakebite.subr A.snakebite/ssh_config_ext A.snakebite/ssh_known_hosts U .snakebite Checked out revision 58. Created link for 'sb'. Created link for 'sbx'. Created link for 'sby'. Created link for 'sbctl'. Fixed permissions for /Users/Trent/.snakebite/snakebite.subr. The following commands can now be executed directly: sb sbx sby sbctl Testing connection.done. Getting a list of your projects...done. Looking up your username for project 'cpython'...done. Getting project details for 'trent.nelson@cpython'...done. Setting current project to 'cpython'...done. Trent, you're now configured for cpython. Enjoy! +---+ |Available Hosts| | (Last Update: 2012-09-11 11:08:01Z) | +---+ | Alias | OS | Arch | +---||--+ | a7|AIX 7.1 | Power4 | | d3|DragonFlyBSD 3.0.2 | x64| |d3x|DragonFlyBSD 3.0.2 | x86| | h2|HP-UX 11iv2 | PA-RISC | | h3|HP-UX 11iv3 | Itanium2 | | i6|IRIX 6.5.30 | MIPS | |n51|NetBSD 5.1.2| x64| | n51x|NetBSD 5.1.2| x86| | o51x|OpenBSD 5.1 | x86| |o51|OpenBSD 5.1 | x64| |s10|Solaris 10 | SPARC | | s9|Solaris 9 | SPARC | +---+ Enter alias: Simply enter any of the aliases in the table and it'll ssh you into that box as cpython@, i.e.: Enter alias: a7 AIX arsenic 1 7 000BF95F4C00 :::. :::...::. .,:: :::.:::. :::.,-: ;;`;;`` ;;;`` `, `;;; ;;; ,;;;'' ,[[ '[[, [[[,/[[[' '[==/, [[ [. '[[ [[[ [[[ c$$$cc$$$c $$c '''$ $$ $$$ Y$c$$ $$$ $$$ 888 888, 888b 88bo, 88bdP 888oo,__ 888Y88 888 `88bo,__,o, YMM ` W YMmMY YUMMM MMM YM MMMYUMP AIX 7.1 IBM IntelliStation 9114-275 2 x 1.4GHz Power4 CPUs 2 x 2Gbps LP9802 FC HBAs 8GB RAM, 4 x 36GB [cpython@arsenic]~% General notes: - Almost all of the hosts have a corresponding cpython build slave, which always lives in ~/buildslave. - You're more than welcome to set up local builds on each box. Keep everything in ~/hg. Some hosts already have a ~/hg dir, others don't. The layout should be: ~/hg/3.x ~/hg/3.2 ~/hg/2.7 If they don't exist, feel free to create them. It's going to be easiest to just clone the corresponding build directory from ~/buildslave, i.e. if you want a local 3.x area but no ~/hg/3.x exists: % cd ~/hg % hg clone ~/buildslave/3.x-*/build 3.x Once a base repo has been created, you can clone a local copy: hg clone 3.x 3.x.trent.issue2811 Try follow that naming convention as it'll make it easier for other developers to figure out what each directory is for. Also, try and keep tabs on local builds and remove things you don't need once you're done. I haven't finished hooking up the SAN yet so everything is on local disks at the moment; disk space is a bit light in some places. - If you're not used to vi shell key bindings, you're going to have a bad time :-) - Almost all of the hosts (except for the *BSDs) have been set up to use a common ~/.zsh and ~/.vim: http:/svn.snakebite.net/home/trunk/ http:/svn.snakebite.net/skel/trunk/ They're both based on random dotfile hacking I've done over the years and are far from elegant -- so, suggestions welcome. If I'm awake and working, I'll be on #python-dev, so that'll be the best place to get me if you need immediate assistance. So, log in and have a play around! Oh, X11 forwarding works, too, just invoke `sbx` (or `sby`) instead of `sb` and it'll invoke ssh with -X or -Y respectively. All the proprietary UNIX hosts have X11 installed, complete with glorious circa-late-nineties Motif eye candy. For those looking for tangible things to do... take a look at the current buildslaves with [SB] in the name -- almost all of them are failing in some way/shape/form, so there's plenty of stuff to get your teeth stuck into :-) Trent. ___ Python-Dev mailing list Python-Dev@python.org
Re: [Python-Dev] Snakebite v0.1: ready for beta testing.
On Tue, Sep 11, 2012 at 05:20:01AM -0700, Trent Nelson wrote: Quick start: % cd ~ svn co http://svn.snakebite.net/.snakebite cd .snakebite sh snakebite.subr For those that already have ~/.snakebite, one of these will work: - sbctl hard-reset, or - svn update sh snakebite.subr In general, from now on (I tweaked it quite a bit this morning), 'sb' automatically updates itself as necessary. (Pro-tip though: `sh snakebite.subr` will always do its best to re-initialize everything, like fixing permissions when svn update strips them, etc.) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Snakebite v0.1: ready for beta testing.
On Tue, Sep 11, 2012 at 05:23:34AM -0700, Brett Cannon wrote: Very cool, Trent! I also love the retro use of svn as a tie-in to how long you have been fighting to bring this project to fruition. =) Haha. I probably shouldn't mention that I started writing all the wrapper .snakebite/svn stuff from scratch last Wednesday :P Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Snakebite v0.1: ready for beta testing.
On Tue, Sep 11, 2012 at 06:16:52AM -0700, Nick Coghlan wrote: On Tue, Sep 11, 2012 at 10:28 PM, Trent Nelson tr...@snakebite.org wrote: (Pro-tip though: `sh snakebite.subr` will always do its best to re-initialize everything, like fixing permissions when svn update strips them, etc.) That's actually causing some problems - if SELinux security context info or ACLs are stored for the directory, then the extra character (either '.' or '+') in the ls -l output causes the permissions check to fail. Easy enough to fix though, the __get_perms command just needs to be changed to: ls -l $1 | cut -b 1-10 Ta, fixed. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tru64 support
On Tue, Sep 04, 2012 at 02:24:10AM -0700, Andrew Svetlov wrote: Is it still up to date? Bug has been created in 2004. I don't see Tru64 in list of available buildbots. Do we need to care about this platform? And how to make sure what existing code works fine for that? Or maybe we can drop support for Tru64 as well as we already does it for, say, VMS? The ES40 I've got sitting in the corner waiting for a rainy day just died a little inside. (HP was also kind enough to donate unlimited Tru64 media/licenses a while back.) As for whether or not it should be a platform CPython cares about -- I'd argue it falls in the same boat as IRIX/MIPS: not so much. I'll be damned if Snakebite doesn't have Tru64/Alpha representation though :-) OpenVMS is an interesting one though; it's still actively supported by HP on Itanium. I've got an Itanium earmarked marked for OpenVMS for an even rainier day, but I wouldn't know OpenVMS is it smacked me in the head, so, not a high priority at the moment. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Snakebite build slaves and developer SSH/GPG public keys
On Fri, Aug 24, 2012 at 07:27:09AM -0700, C. Titus Brown wrote: On Thu, Aug 23, 2012 at 11:09:14AM +0200, Giampaolo Rodol? wrote: For committers on other Python projects like Buildbot, Django and Twisted that may be reading this -- yes, the plan is to give you guys Snakebite access/slaves down the track too. I'll start looking into that after I've finished setting up the remaining slaves for Python. (Setting up a keys repo will definitely help (doesn't have to be hg -- feel free to use svn/git/whatever, just try and follow the same layout).) This is so great! I've been looking forward to this for a long time and kept visiting the site every now and then to see if there was any progress. I'd surely use this for psutil if you'll let me. Also, at some point I would suggest to introduce the possibility to donate some money in order to help supporting what I think must be a pretty complex infrastructure requiring a lot of resources, both in terms of hardware and time/labor. Don't forget the heavy Xanax requirements on the part of the technical owner of the space. Dunno if Trent will put up any pictures but I'm dreading the day that building maintenance gives me a call asking what the heck I've done to the windows, drains, and power. What's an industrial exhaust fan or two bolted to a window frame between friends eh. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Snakebite build slaves and developer SSH/GPG public keys
On Fri, Aug 24, 2012 at 07:39:54AM -0700, Trent Nelson wrote: On Fri, Aug 24, 2012 at 07:27:09AM -0700, C. Titus Brown wrote: Don't forget the heavy Xanax requirements on the part of the technical owner of the space. Dunno if Trent will put up any pictures but I'm dreading the day that building maintenance gives me a call asking what the heck I've done to the windows, drains, and power. What's an industrial exhaust fan or two bolted to a window frame between friends eh. http://i.imgur.com/BbKn9.jpg that wouldn't be very effective if I left the window panes in now, would it? ;-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Infrastructure] Snakebite build slaves and developer SSH/GPG public keys
On Thu, Aug 23, 2012 at 12:24:33AM -0700, Martin v. Löwis wrote: On 23.08.2012 02:43, R. David Murray wrote: On Thu, 23 Aug 2012 10:53:34 +1200, Noah Kantrowitz n...@coderanger.net wrote: For everyone with a record in the Chef server (read: everyone with SSH access to any of the PSF servers at OSL) I can easily give you automated access. Whats the easiest format? I can give you a Python script that will spit out files or JSON or more or less whatever else you want. That isn't going to be the right set of keys for Trent's purposes (though it is likely to be a subset). The keyfile we use for the hg repository is. ... for which it would be easiest if we give Trent access to the repository storing these keys. I'm a bit hesitant to put public keys into the real world-wide public, given the past history of easily-breakable public keys. For PGP, this is less of a concern than for SSH, since the threats are smaller (plus users where aware that they might have to publish the key when they created it). Hmmm. So, from my perspective, I have the following goals: - Commit access to ssh://hg.python.org implies access to `ssh cpython@snakebite` via the exact same key. - No extra administrative overhead/burden on infrastructure@ (with regards to ssh key management, i.e. an entry in .ssh/ authorized_keys should be sufficient for implicit snakebite access). Factoring in your (valid) security concerns, here's my altered proposal: - Let's just call the repo 'snakebite', and have it accessible only to ssh committers, no public http access. Calling it something generic like 'keys' may invite phantom requirements like being able to store multiple identities/keys etc. I don't need that for snakebite; one ssh key and one optional gpg key is all I want :-) - Same repo layout as before -- GPG keys not required unless I need to send you something encrypted via e-mail (RDP is the only use case I can think of for this). - I'll whip up the glue to take our current .ssh/authz and dump it into the 'snakebite' repo. We can refine that process down the track (with automation and whatnot). If there are no objections I can take this offline with inf@. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Snakebite build slaves and developer SSH/GPG public keys
On Thu, Aug 23, 2012 at 02:09:14AM -0700, Giampaolo Rodolà wrote: For committers on other Python projects like Buildbot, Django and Twisted that may be reading this -- yes, the plan is to give you guys Snakebite access/slaves down the track too. I'll start looking into that after I've finished setting up the remaining slaves for Python. (Setting up a keys repo will definitely help (doesn't have to be hg -- feel free to use svn/git/whatever, just try and follow the same layout).) This is so great! I've been looking forward to this for a long time and kept visiting the site every now and then to see if there was any progress. I'd surely use this for psutil if you'll let me. Hey, psutil, I use that! :-) Awesome candidate for Snakebite, too, given the unavoidable platform-specific C code behind the scenes. I'll take it up with you offline. Also, at some point I would suggest to introduce the possibility to donate some money in order to help supporting what I think must be a pretty complex infrastructure requiring a lot of resources, both in terms of hardware and time/labor. You have no idea :-/ Regards, Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Snakebite build slaves and developer SSH/GPG public keys
On Wed, Aug 22, 2012 at 03:28:16PM -0700, Trent Nelson wrote: Hi folks, I've set up a bunch of Snakebite build slaves over the past week. One of the original goals was to provide Python committers with full access to the slaves, which I'm still keen on providing. What's a nice simple way to achieve that in the interim? Quick update: Martin's hooked me up with everything I need for now. I'll send out another e-mail once I've set up the necessary glue on my end. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Snakebite build slaves and developer SSH/GPG public keys
Hi folks, I've set up a bunch of Snakebite build slaves over the past week. One of the original goals was to provide Python committers with full access to the slaves, which I'm still keen on providing. What's a nice simple way to achieve that in the interim? Here's what I was thinking: - Create a new hg repo: hg.python.org/keys. - Committers can push to it just like any other repo (i.e. same ssh/authz configuration as cpython). - Repo is laid out as follows: keys/ python username/ ssh (ssh public key) gpg (gpg public key) - Prime the repo with the current .ssh/authorized_keys (presuming you still use the --tunnel-user facility?). That'll provide me with everything I need to set up the relevant .ssh/authorized_keys stuff on the Snakebite side. GPG keys will be handy if I ever need to send passwords over e-mail (which I'll probably have to do initially for those that want to RDP into the Windows slaves). Thoughts? As for the slaves, here's what's up and running now: - AMD64 Mountain Lion [SB] - AMD64 FreeBSD 8.2 [SB] - AMD64 FreeBSD 9.1 [SB] - AMD64 NetBSD 5.1.2 [SB] - AMD64 OpenBSD 5.1 [SB] - AMD64 DragonFlyBSD 3.0.2 [SB] - AMD64 Windows Server 2008 R2 SP1 [SB] - x86 NetBSD 5.1.2 [SB] - x86 OpenBSD 5.1 [SB] - x86 DragonFlyBSD 3.0.2 [SB] - x86 Windows Server 2003 R2 SP2 [SB] - x86 Windows Server 2008 R2 SP1 [SB] All the FreeBSD ones use ZFS, all the DragonFly ones use HAMMER. DragonFly, NetBSD and OpenBSD are currently reporting all sorts of weird and wonderful errors, which is partly why I want to set up ssh access sooner rather than later. Other slaves on the horizon (i.e. hardware is up, OS is installed): - Windows 8 x64 (w/ VS2010 and VS2012) - HP-UX 11iv2 PA-RISC - HP-UX 11iv3 Itanium (64GB RAM) - AIX 5.3 RS/6000 - AIX 6.1 RS/6000 - AIX 7.1 RS/6000 - Solaris 9 SPARC - Solaris 10 SPARC Nostalgia slaves that probably won't ever see green: - IRIX 6.5.33 MIPS - Tru64 5.1B Alpha If anyone wants ssh access now to the UNIX platforms in order to debug/test, feel free to e-mail me directly with your ssh public keys. For committers on other Python projects like Buildbot, Django and Twisted that may be reading this -- yes, the plan is to give you guys Snakebite access/slaves down the track too. I'll start looking into that after I've finished setting up the remaining slaves for Python. (Setting up a keys repo will definitely help (doesn't have to be hg -- feel free to use svn/git/whatever, just try and follow the same layout).) Regards, Trent that-took-a-bit-longer-than-expected Nelson. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Mountain Lion drops sign of zero, breaks test_cmath...
The Mountain Lion build slave I set up earlier this evening fails on
test_cmath:
==
FAIL: test_specific_values (test.test_cmath.CMathTests)
--
Traceback (most recent call last):
File
/Volumes/bay2/buildslave/cpython/2.7.snakebite-mountainlion-amd64/build/Lib/test/test_cmath.py,
line 352, in test_specific_values
msg=error_message)
File
/Volumes/bay2/buildslave/cpython/2.7.snakebite-mountainlion-amd64/build/Lib/test/test_cmath.py,
line 94, in rAssertAlmostEqual
'got {!r}'.format(a, b))
AssertionError: atan: atan(complex(0.0, 0.0))
Expected: complex(0.0, 0.0)
Received: complex(0.0, -0.0)
Received value insufficiently close to expected value.
Mountain Lion's atan/log1p appear to drop the negative sign when
passed in -0.0, whereas previous versions of OS X didn't:
Mountain Lion:
% ~/log1p-viper
log1p_drops_zero_sign_test:
atan2(log1p(-0.), -1.) != atan2(-0., -1.)
3.14159 vs -3.14159
atan_drops_zero_sign_test:
atan2(-0., 0.): -0.0
atan2( 0., -0.): 3.14159
atan2(-0., -0.): -3.14159
atan2( 0., 0.): 0.0
log1p(-0.): 0.0
log1p( 0.): 0.0
Lion:
% ./log1p
log1p_drops_zero_sign_test:
atan2(log1p(-0.), -1.) == atan2(-0., -1.)
-3.14159 vs -3.14159
atan_drops_zero_sign_test:
atan2(-0., 0.): -0.0
atan2( 0., -0.): 3.14159
atan2(-0., -0.): -3.14159
atan2( 0., 0.): 0.0
log1p(-0.): -0.0
log1p( 0.): 0.0
(The C code for that is below.)
configure.ac already has a test for this (it makes mention of AIX
having similar behaviour), and the corresponding sysconfig entry
named 'LOG1P_DROPS_ZERO_SIGN' is already being used on a few tests,
i.e.:
# The algorithm used for atan and atanh makes use of the system
# log1p function; If that system function doesn't respect the sign
# of zero, then atan and atanh will also have difficulties with
# the sign of complex zeros.
@requires_IEEE_754
@unittest.skipIf(sysconfig.get_config_var('LOG1P_DROPS_ZERO_SIGN'),
system log1p() function doesn't preserve the sign)
def testAtanSign(self):
for z in complex_zeros:
self.assertComplexIdentical(cmath.atan(z), z)
@requires_IEEE_754
@unittest.skipIf(sysconfig.get_config_var('LOG1P_DROPS_ZERO_SIGN'),
system log1p() function doesn't preserve the sign)
def testAtanhSign(self):
for z in complex_zeros:
self.assertComplexIdentical(cmath.atanh(z), z)
Taking a look at cmath_testcases.txt, and we can see this:
-- These are tested in testAtanSign in test_cmath.py
-- atan atan 0.0 0.0 - 0.0 0.0
-- atan0001 atan 0.0 -0.0 - 0.0 -0.0
-- atan0002 atan -0.0 0.0 - -0.0 0.0
-- atan0003 atan -0.0 -0.0 - -0.0 -0.0
However, a few lines down, those tests crop up again:
-- special values
atan1000 atan -0.0 0.0 - -0.0 0.0
snip
atan1014 atan 0.0 0.0 - 0.0 0.0
which is what causes the current test failures. I hacked
test_cmath.py a bit to spit out all the errors it finds after
it's finished parsing the test file (instead of bombing out on
the first one), and it yielded this:
FAIL: test_specific_values (test.test_cmath.CMathTests)
--
Traceback (most recent call last):
File
/Volumes/bay2/buildslave/cpython/3.2.snakebite-mountainlion-amd64/build/Lib/test/test_cmath.py,
line 446, in test_specific_values
self.fail(\n.join(failures))
AssertionError: atan1000: atan(complex(-0.0, 0.0))
Expected: complex(-0.0, 0.0)
Received: complex(-0.0, -0.0)
Received value insufficiently close to expected value.
atan1014: atan(complex(0.0, 0.0))
Expected: complex(0.0, 0.0)
Received: complex(0.0, -0.0)
Received value insufficiently close to expected value.
atanh0225: atanh(complex(-0.0, 5.6067e-320))
Expected: complex(-0.0, 5.6067e-320)
Received: complex(0.0, 5.6067e-320)
Received value insufficiently close to expected value.
atanh0227: atanh(complex(-0.0, -3.0861101e-316))
Expected: complex(-0.0, -3.0861101e-316)
Received: complex(0.0, -3.0861101e-316)
Received value insufficiently
Re: [Python-Dev] Mountain Lion drops sign of zero, breaks test_cmath...
On Fri, Aug 17, 2012 at 06:50:09AM -0700, Stefan Krah wrote: R. David Murray rdmur...@bitdance.com wrote: --- a/Lib/test/test_cmath.pyThu Aug 16 22:14:43 2012 +0200 +++ b/Lib/test/test_cmath.pyFri Aug 17 07:54:05 2012 + Open an issue on the tracker and make mark.dickinson (and maybe skrah) nosy. I think this issue covers the problem: http://bugs.python.org/issue15477 Ah! I'll update that with my notes. (FWIW, I've changed my mind; I think the correct action is to remove the erroneous entries from the test file, rather than patch rAssertAlmostEqual.) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Switching to Visual Studio 2010
On Thu, Jan 26, 2012 at 12:54:31PM -0800, mar...@v.loewis.de wrote: Is this considered a new feature that has to be in by the first beta? I'm hoping to have it completed much sooner than that so we can get mileage on it, but is there a cutoff for changing the compiler? At some point, I'll start doing this myself if it hasn't been done by then, and I would certainly want the build process adjusted (with all buildbots updated) before beta 1. I... I think I might have already done this, inadvertently. I needed an x64 VS2010 debug build of Subversion/APR*/Python a few weeks ago -- forgetting the fact that we're still on VS2008. By the time I got to building Python, I'd already coerced everything else to use VS2010, so I just bit the bullet and coerced Python to use it too, including updating all the buildbot scripts and relevant externals to use VS2010, too. Things that immediately come to mind as potentially being useful: * Three new buildbot scripts: - build-amd64-vs10.bat - clean-amd64-vs10.bat - external-amd64-vs10.bat * Updates to externals/(tcl|tk)-8.5.9.x so that they both build with VS2010. This was a tad fiddly. I ended up creating makefile.vs10 from win/makefile.vc and encapsulating the changes there, then calling that from the buildbot *-vs10.bat scripts. I had to change win/rules.vc, too. * A few other things I can't remember off the top of my head. So, I guess my question is, is that work useful? Based on Martin's original list, it seems to check a few boxes. Brian, what are your plans? Are you going to continue working in hg.python.org/sandbox/vs2010port then merge everything over when ready? I have some time available to work on this for the next three weeks or so and would like to help out. Regards, Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Switching to Visual Studio 2010
On Sun, Jan 29, 2012 at 12:23:14PM -0800, Trent Nelson wrote: * Updates to externals/(tcl|tk)-8.5.9.x so that they both build with VS2010. Before I go updating tcl/tk, any thoughts on bumping our support to the latest revision, 8.5.11? I guess the same question applies to all the externals, actually (zlib, openssl, sqlite, bsddb, etc). In the past we've typically bumped up our support to the latest version prior to beta, then stuck with that for the release's life, right? Semi-related note: is svn.python.org/externals still the primary repo for externals? (I can't see a similarly named hg repo.) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Stable buildbots
On 14-Nov-10 3:48 AM, David Bolen wrote: This is a completely separate issue, though probably around just as long, and like the popup problem its frequency changes over time. By hung here I'm referring to cases where something must go wrong with a test and/or its cleanup such that a python_d process remains running, usually several of them at the same time. My guess: the hung (single-threaded) Python process has called select() without a timeout in order to wait for some data. However, the data never arrives (due to a broken/failed test), and the select() never returns. On Windows, processes seem harder to kill when they get into this state. If I purposely wedge a Windows process via select() via the interactive interpreter, ctrl-c has absolutely no effect (whereas on Unix, ctrl-c will interrupt the select()). As for why kill_python.exe doesn't seem to be able to kill said wedged processes, the MSDN documentation on TerminateProcess[1] states the following: The terminated process cannot exit until all pending I/O has been completed or canceled. (sic) It's not unreasonable to assume a wedged select() constitutes pending I/O, so that's a possible explanation as to why kill_python.exe isn't able to terminate the processes. (Also, kill_python currently assumes TerminateProcess() always works; perhaps this optimism is misplaced. Also note the XXX TODO regarding the fact that we don't kill processes that have loaded our python*.dll, but may not be named python_d.exe. I don't think that's the issue here, though.) On 14-Nov-10 5:32 AM, David Bolen wrote: Martin v. Löwismar...@v.loewis.de writes: This is what kill_python.exe is supposed to solve. So I recommend to investigate why it fails to kill the hanging Pythons. Yeah, I know, and I can't say I disagree in principle - not sure why Windows doesn't let the kill in that module work (or if there's an issue actually running it under all conditions). At the moment though, I do know that using the sysinternals pskill utility externally (which is what I currently do interactively) definitely works so to be honest, That's interesting. (That kill_python.exe doesn't kill the wedged processes, but pskill does.) kill_python is pretty simple, it just calls TerminateProcess() after acquiring a handle with the relevant PROCESS_TERMINATE access right. That being said, that's the recommended way to kill a process -- I doubt pskill would be going about it any differently (although, it is sysinternals... you never know what kind of crazy black magic it's doing behind the scenes). Are you calling pskill with the -t flag? i.e. kill process and all dependents? That might be the ticket, especially if killing the child process that wedged select() is waiting on causes it to return, and thus, makes it killable. Otherwise, if it happens again, can you try kill_python.exe first, then pskill, and confirm if the former fails but the latter succeeds? Trent. [1]: http://msdn.microsoft.com/en-us/library/ms686714(VS.85).aspx ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Snakebite, buildbot and low hanging fruit -- feedback wanted! (Was Re: SSH access against buildbot boxes)
On 07-Nov-10 1:55 AM, Nick Coghlan wrote: On Sun, Nov 7, 2010 at 3:53 AM, Giampaolo Rodolàg.rod...@gmail.com wrote: In such cases I would find more easy to be able to connect to the machine and test myself rather than create a separate branch, commit, schedule a buildbot run, wait for it to complete and see whether everything is green. On the other side I perfectly understand how opening up blanket ssh access is not something everyone is comfortable with doing. AFAICR there was someone who was setting up an evironment to solve exactly this problem but I'm not sure whether this is already usable. Dealing with exactly this problem is one of the goals of the Snakebite project. As far as I know, the folks behind that project are still working on it - I've cc'ed Trent Nelson to see if he can provide any additional info on the topic. Thanks for the ping Nick, I might have missed this otherwise. Good timing, too, as Titus and I were just discussing which low hanging fruit/pain points Snakebite should tackle first (now that all the server room stuff has finally been taken care of). Luckily, the problems that we faced 2.5 years ago when I came up with the idea of Snakebite are still just as ever present today ;-) 1. Not having access to buildbots is a pain when something doesn't work right. Doing dummy debug commits against trunk to try and coerce some more information out of a failing platform is painful. Losing a build slave entirely due to a particularly hard crash and requiring the assistance of the owner is also super frustrating. 2. The buildbot web interface for building non-(trunk|2.x|py3k) branches is also crazy unfriendly. Per-activity branches are a great way to isolate development, even with Subversion, but it kinda' blows that you don't *really* get any feedback about how your code behaves on other platforms until you re-integrate your changes back into a mainline branch. (I'm sure none of us have been masochistic enough to manually kick off individual builds for every platform via the buildbot web page after every commit to a non-standard branch.) So, enter Snakebite. We've got three racks filled with way more hardware than I should have ever purchased. Ignoring the overhead of having to set machines up and whatnot, let's just assume that over the next couple of months, if there's a platform we need a stable buildbot for, Snakebite can provide it. (And if we feel like bringing IRIX/MIPS and Tru64/Alphas back as primary platforms, we've got the hardware to do that, too ;-).) Now, the fact that they're all in the one place and under my complete control is a big advantage, as I can start addressing some of the pain points that lead me down this twisted path 2.5 years ago. I'd like to get some feedback from the development community on what they'd prefer. In my mind, I could take one of the following two steps: 1. Set up standard build slaves on all the platforms, but put something in place that allowed committers to ssh/mstsc in to said slaves when things go wrong in order to aid with debugging and/or maintaining general buildbot health (OK'ing modal crash dialogues on Windows, for example). 2. Address the second problem of the buildbot web interface sucking for non-standard branches. I'm thinking along the lines of a hack to buildbot, such that upon creation of new per-activity branches off a mainline, something magically runs in the background and sets up a complete buildbot view at python.snakebite.org/dev/buildbot/your-branch-name, just as if you were looking at a trunk buildbot page. I'm not sure how easy the second point will be when we switch to hg; and I'll admit if there have been any python-dev discussions about buildbot once we're on hg, I've missed them. Of course there's a third option, which is using the infrastructure I've mentioned to address a similarly annoying pain point I haven't thought of -- so feel free to mention anything else you'd like to see first instead of the above two things. Titus, for example, alluded to some nifty way for a committer to push his local hg branch/changes somewhere, such that it would kick off builds on multiple platforms in the same sorta' vein as point 2, but able to leverage cloud resources like Amazon's EC2, not just Snakebite hardware. Look forward to hearing some feedback! Regards, Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IBM P-690 server looking for a home
On 19-Aug-10 10:48 AM, Randall Walls wrote: Greetings, The company I work for has an IBM P-690 server that is in the process of being retired. It is still a viable server, and has seen almost 0 use (it was our failover machine). Unfortunately for us, this machine has little to no resale value, and will probably be junked. I'd rather it go to a good home, and having taken advantage of the work of the python development community for a number of years (we use python extensively in system admin and database work), I saw this as an opportunity to give back a little. So, If anyone is interested in this machine, please let me know. We are looking at perhaps a November time frame for when it will be removed from our remote site. The P690 is no small machine, it is the size of a full rack and has 32 Power4 processors in it and takes (I believe) 2 or 3 phase 220 Volt power. It weighs nearly a ton. We are running AIX5.3 on it, but I believe that the machine is capable of running a PowerPC flavor of Linux as well. This would make a great test machine for python HPC modules or as a community box where developers could test their code against a PowerPC architecture. It has lots of life left and I'd rather see it put to use then thrown away. Snakebite[1]'s always got an eye out for free hardware, but dang, that's one chunky piece of kit. I'll follow up in private. (And yeah, I'm still working on Snakebite, for those that are interested. Turns out hosting three racks of heavy-duty hardware in the corner room of a (graciously donated) science lab takes a bit longer than originally anticipated. Who would have thought.) Regards, Trent no-news-is-good-news Nelson. [1]: http://www.snakebite.org/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python on Windows with CoApp
Howdy folks, Quick e-mail at 34,000ft (aren't wifi-enabled flights great?) to mention a new initiative that's been started by Microsoft called CoApp (Common Opensource Application Publishing Platform). The aim is simple: make open source software rock on Windows ;-) It's probably easiest to think of it as a Microsoft-endorsed-but-community-run open source distribution for Windows, akin to all the various package managers for Linux distributions and ports/packages for the *BSDs. There are specific user and developer experiences we'll be addressing -- like making it easy to install and use open source software, or use it within your own project (open source or not). CoApp will affect Python in one of two ways. Once there's a clear-cut specification for open source projects to follow, Python can either decide to follow it, or not. The same applies to all open source packages, actually. For those that follow it, great! If not, no problem -- the plan is to shallow-fork such projects via launchpad and the CoApp community will take responsibility for getting releases of open source projects into CoApp shape. It's in its infancy at the moment -- it took the chap (Garrett Serack) who's spearheading it at Microsoft about six months to get it all signed off by the lawyers and platform/server VPs. So, for those of you out there who are Windows-inclined, now's a perfect time to get involved to help shape the direction of CoApp going forward. The website/wiki is http://coapp.org/ and the launchpad project site is http://launchpad.net/coapp (which is where the mailing list is hosted). We're actually having a 'CoApp Development Summit' tomorrow and Friday in Seattle (that Microsoft's graciously sponsored). The event will be accessible via Live Meeting for those that are interested: http://coapp.org/Project_Planning/CoApp_Design_and_Development_Summit Regards, Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 7 updated
Does anyone know of a way to teach vim that C sources in a python checkout should have 4-space indents without changing the defaults for other C files? I use this in my vimrc: indentation: use detectindent plugin if possible set autoindent set smartindent try let g:detectindent_preferred_expandtab = 1 let g:detectindent_preferred_tabsize = 8 let g:detectindent_preferred_indent = 4 source $VIMRUNTIME/plugin/detectindent.vim au BufNewFile,BufRead * .* DetectIndent catch set smarttab set expandtab set tabstop=8 set shiftwidth=4 set softtabstop=4 set textwidth=80 endtry *** And this is plugin/detectindent.vim: Name: detectindent (global plugin) Version: 1.0 Author:Ciaran McCreesh ciaranm at gentoo.org Updates: http://dev.gentoo.org/~ciaranm/vim/ Purpose: Detect file indent settings License: You may redistribute this plugin under the same terms as Vim itself. Usage: :DetectIndent to prefer expandtab to noexpandtab when detection is impossible: :let g:detectindent_preferred_expandtab = 1 to set a preferred indent level when detection is impossible: :let g:detectindent_preferred_indent = 4 Requirements: Untested on Vim versions below 6.2 fun! SIDIsCommentStart(line) comments isn't reliable if ft == c || ft == cpp return -1 != match(a:line, '/\*') else return 0 endif endfun fun! SIDIsCommentEnd(line) if ft == c || ft == cpp return -1 != match(a:line, '\*/') else return 0 endif endfun fun! SIDDetectIndent() let l:has_leading_tabs= 0 let l:has_leading_spaces = 0 let l:shortest_leading_spaces_run = 0 let l:longest_leading_spaces_run = 0 let l:idx_end = line($) let l:idx = 1 while l:idx = l:idx_end let l:line = getline(l:idx) try to skip over comment blocks, they can give really screwy indent settings in c/c++ files especially if SIDIsCommentStart(l:line) while l:idx = l:idx_end ! SIDIsCommentEnd(l:line) let l:line = getline(l:idx) let l:idx = l:idx + 1 endwhile let l:idx = l:idx + 1 continue endif let l:leading_char = strpart(l:line, 0, 1) if l:leading_char == \t let l:has_leading_tabs = 1 elseif l:leading_char == only interested if we don't have a run of spaces followed by a tab. if -1 == match(l:line, '^ \+\t') let l:has_leading_spaces = 1 let l:spaces = strlen(matchstr(l:line, '^ \+')) if l:shortest_leading_spaces_run == 0 || \ l:spaces l:shortest_leading_spaces_run let l:shortest_leading_spaces_run = l:spaces endif if l:spaces l:longest_leading_spaces_run let l:longest_leading_spaces_run = l:spaces endif endif endif let l:idx = l:idx + 1 endwhile if l:has_leading_tabs ! l:has_leading_spaces tabs only, no spaces set noexpandtab if exists(g:detectindent_preferred_tabsize) let shiftwidth = g:detectindent_preferred_indent let tabstop = g:detectindent_preferred_indent endif elseif l:has_leading_spaces ! l:has_leading_tabs spaces only, no tabs set expandtab let shiftwidth = l:shortest_leading_spaces_run elseif l:has_leading_spaces l:has_leading_tabs spaces and tabs set noexpandtab let shiftwidth = l:shortest_leading_spaces_run , time to guess how big tabs are if l:longest_leading_spaces_run 2 let tabstop = 2 elseif l:longest_leading_spaces_run 4 let tabstop = 4 else let tabstop = 8 endif else no spaces, no tabs if exists(g:detectindent_preferred_tabsize) let shiftwidth = g:detectindent_preferred_indent let tabstop = g:detectindent_preferred_indent endif if exists(g:detectindent_preferred_expandtab) set expandtab endif endif endfun command! -nargs=0 DetectIndent call SIDDetectIndent() ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Broken link to download (Mac OS X)
What happened to the big-ass computer farm for Python which was being put together by someone at (I think) Michigan State? That sounds a lot like Snakebite (www.snakebite.org), which is still... uhhh, a work in progress ;-) We've run into an issue recently that's thwarted progress, but that'll hopefully be resolved in the next couple of weeks. And then... full steam ahead! Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Broken link to download (Mac OS X)
What happened to the big-ass computer farm for Python which was being put together by someone at (I think) Michigan State? That sounds a lot like Snakebite (www.snakebite.org), which is still... uhhh, a work in progress ;-) Actually, for those that are interested, here's a copy of the presentation I gave at the Testing in Python session at PyCon a few months ago: http://www.snakebite.org/presentations/snakebite-pycon2010-tip.pptx (Office 2007-2010) http://www.snakebite.org/presentations/snakebite-pycon2010-tip.ppt (Office 97-2003) If anything, it'll shed some light on all the unforeseen issues we've been running into since the project's inception. The presentation is a little out of date -- I spent three months earlier this year on the network and it's definitely in the most respectable state it's been in yet. Coupla' photos for those that are interested: http://snakebite.org/images/IMG_4384.JPG http://snakebite.org/images/IMG_4392.JPG http://snakebite.org/images/IMG_4393.JPG http://snakebite.org/images/IMG_4394.JPG http://snakebite.org/images/IMG_4395.JPG http://snakebite.org/images/IMG_4396.JPG http://snakebite.org/images/IMG_4401.JPG http://snakebite.org/images/IMG_4402.JPG http://snakebite.org/images/IMG_4403.JPG http://snakebite.org/images/IMG_4405.JPG http://snakebite.org/images/IMG_4410.JPG http://snakebite.org/images/IMG_4418.JPG http://snakebite.org/images/IMG_4424.JPG http://snakebite.org/images/IMG_4425.JPG We've got three racks filled to the brim with all sorts of servers: - 4xItanium 2 @ 1.5GHz, 16GB RAM, HP-UX 11iv3 - 4xItanium 2 @ 1.5GHz, 30GB RAM, RHEL 5.3 - 2xUltraSPARC III 900MHz, 8GB, Solaris 10 (file/zfs/nfs server -- 16x146GB 2Gb FC) - 2xUltraSPARC III 1.2GHz, 4GB, Solaris 10 - 2xPA-RISC 875MHz, 8GB, HP-UX 11iv1 - 4 AIX boxes w/ 2x1.5GHz, 8GB, AIX 5.1, 5.2, 5.3 6.1 - 10 dedicated VMware x86/64 boxes, ranging from dual core 8GB to 8 core monsters with 64GB - 4x667MHz AlphaServer, 8GB, Tru64 - 4x600MHz SGI Octane 300, IRIX 6.22 - and lots of other stuff. Actually, the only platform we don't have is Mac OS X. Although I've got a contact at Apple that I'll start harassing again once I'm back in East Lansing. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py_ssize_t support for ctypes arrays and pointers
On Fri, Mar 20, 2009 at 08:00:46PM +0100, Thomas Heller wrote: Since I do not have a machine with so much memory: Does one of the buildbots allow to run tests for this feature, or do I have to wait for the snakebite farm? Will you be at PyCon? The wait might not be as bad as you think ;-) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [snakebite] snakebite for GSoC?
On Thu, Mar 19, 2009 at 10:32:03AM -0700, ajaksu wrote: Does anyone have good ideas for assigning students to snakebite? Is it too early? Perhaps a little too early, python-dev@ won't know anything about Snakebite yet as I haven't publicly announced it there ;-) Watch this space closer to PyCon. FWIW, though, we're planning for Snakebite to be *very* involved with GSoC/GHOP. I think the client-side 'Snakebite daemon' and server-side stuff described at http://tinyurl.com/beyond-buildbot would be great projects. Indeed. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ?spurious? Timeout in BSDDB under MS Windows
On Fri, Sep 05, 2008 at 05:55:13PM +0200, Jesus Cea wrote: Trent, are you available to look at the ?spurious? timeout failures in bsddb replication code in the Windows buildbot?. Ten seconds timeout should be plenty enough. I can't debug any MS Windows issue myself; this is a Microsoft-free environment. I think I added in 10 seconds 'cause the tests kept failing when it was at 2 seconds ;-) I remember digging around the code a bit when I bumped bsddb to 4.7 on Windows to try and figure out what was going on. As far as I could tell it wasn't anything obvious caused by the Python code; is it possible this could be an issue with the underlying bsddb code? Side note: are all your recent bsddb changes that went into trunk also available in py3k? I've had bump py3k Windows buildbots to use bsddb 4.7.25 on my todo list for far too long! Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] what version of bsddb to use/accept for 2.6/3.0
On Sun, Aug 24, 2008 at 06:15:06PM -0700, Neal Norwitz wrote: It looks like the WIndows buildbots use 4.4.20. Unfortunately, the Windows bots aren't in great shape either. It won't be that hard to bump 3.0 on Windows to use 4.7 (assuming that the 3.0 bsddb codebase has Jesus's new code). I'll be able to take a look at this early next week. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Any PEP about 2.6 - 3000 code transition?
(*) slides: http://www.interlink.com.au/anthony/tech/talks/OSCON2008/porting3.pdf Hilarious! Seems like that would have been a riot of a session to attend. (I'm kicking myself for attending some other uninteresting talk when yours was on.) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_multiprocessing: test_listener_client flakiness
I just found the cause of the problem ten minutes ago:
It seems that when a socket listens on the address 127.0.0.1 or
localhost, another process cannot connect to it using the machine's
name (even from the same machine).
The best seems to listen with the empty address .
That doesn't seem right. If that were the case, none of the hundreds of
network-oriented tests would work either. These all bind against ('localhost',
0) by default which seems to Do The Right Thing on all the platforms we have
buildbots for.
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_multiprocessing:test_listener_client flakiness
Well, on my win2k machine getfqdn('127.0.0.1') returns the actual name
of the machine. This is the name that was stored in server.address.
Hence my patch that simply remove the call to getfqdn.
+1 to ditching getfqdn, following patch fixes the issue on my buildbot server:
Index: connection.py
===
--- connection.py (revision 64369)
+++ connection.py (working copy)
@@ -215,10 +215,7 @@
self._socket = socket.socket(getattr(socket, family))
self._socket.bind(address)
self._socket.listen(backlog)
-address = self._socket.getsockname()
-if type(address) is tuple:
-address = (socket.getfqdn(address[0]),) + address[1:]
-self._address = address
+self._address = self._socket.getsockname()
self._family = family
self._last_accepted = None
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_multiprocessing:test_listener_client flakiness
This is a common problem. Binding to '127.0.0.1' will bind to *only*
that address;
Indeed.
binding to will bind to *all* addresses the machine
is known by.
Agreed again. I believe what we're dealing with here though is a lack of
clarity regarding what role the 'address' attribute exposed by
multiprocess.connection.Listener should play. The way test_listener_client()
is written, it effectively treats 'address' as an end-point that can be
connected to directly (irrespective of the underlying family (i.e. AF_INET,
AF_UNIX, AF_PIPE)).
I believe the problems we've run into stem from the fact that the API doesn't
provide any guarantees as to what 'address' represents. The test suite assumes
it always reflects a connectable end-point, which I think is more than
reasonable. Unfortunately, nothing stops us from breaking this invariant by
constructing the object as Listener(family='AF_INET', address=('0.0.0.0', 0)).
How do I connect to an AF_INET Listener (i.e. SocketListener) instance whose
'address' attribute reports '0.0.0.0' as the host? I can't.
So, for now, I think we should enforce this invariant by raising an exception
in Listener.__init__() if self._socket.getsockbyname()[0] returns '0.0.0.0'.
In effect, tightening up the API such that we can guarantee Listener.address
will always represent a connectable end-point. We can look at how to service
'listen on all available interfaces' semantics at a later date -- that adds far
less value IMO than being able to depend on the said guarantee.
Thoughts?
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] test_multiprocessing: test_listener_client flakiness
I gave my Windows buildbots a little bit of TLC last night. This little
chestnut in test_multiprocessing.py around line 1346 is causing my buildbots to
wedge more often than not:
def test_listener_client(self):
for family in self.connection.families:
l = self.connection.Listener(family=family)
p = self.Process(target=self._test, args=(l.address,))
p.set_daemon(True)
p.start()
conn = l.accept()
self.assertEqual(conn.recv(), 'hello')
p.join()
l.close()
The wedging will be a result of that accept() call. Not knowing anything about
the module or the test suite, I can only assume that there's a race condition
introduced between when the subprocess attempts to connect to the listener,
versus when the l.accept() call is actually entered. (On the basis that a race
condition would explain why sometimes it wedges and sometimes it doesn't.)
Just FYI, the error in the buildbot log
(http://www.python.org/dev/buildbot/all/x86%20W2k8%20trunk/builds/810/step-test/0)
when this occurs is as follows:
test_multiprocessing
command timed out: 1200 seconds without output
SIGKILL failed to kill process
using fake rc=-1
program finished with exit code -1
remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most
recent call last):
Failure: buildbot.slave.commands.TimeoutError: SIGKILL failed to kill process
]
(The fact it can't be killed cleanly is a bug in Twisted's
signalProcess('KILL') method, which doesn't work against Python processes that
have entered accept() calls on Windows (which present the 'wedged' behaviour
and have to be forcibly killed with OpenProcess/TerminateProcess).)
Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_multiprocessing:test_listener_client flakiness
The server component *is* listening to localhost by default. The client in the code I pasted has the server's .address attribute passed into it to tell the client where to connect to. Interestingly enough, I can't coerce test_multiprocessing.test_listener_client to wedge on my XP laptop. When I run it on my Windows Server 2008 box (same box the buildbots run on), it wedges every time. So, I suspect this isn't actually a race like I first thought, but something quirky with 2008. Will dig around for a bit longer and see what I can find... Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
