Re: [Python-Dev] PEP 3118: Extended buffer protocol (new version)
Carl Banks wrote: The thing that bothers me about this whole flags setup is that different flags can do opposite things. Some of the flags RESTRICT the kind of buffers that can be exported (Py_BUF_WRITABLE); other flags EXPAND the kind of buffers that can be exported (Py_BUF_INDIRECT). That is highly confusing and I'm -1 on any proposal that includes both behaviors. (Mutually exclusive sets of flags are a minor exception: they can be thought of as either RESTICTING or EXPANDING, so they could be mixed with either.) The mutually exclusive set is the one example of the restriction that you gave. I think the flags setup I've described is much closer to your Venn diagram concept than you give it credit for. I've re-worded some of the discussion (see http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/numpy/doc/pep_buffer.txt ) so that it is more clear that each flag is a description what kind of buffer the consumer is prepared to deal with. For example, if the consumer cares about what's 'in' the array, it uses Py_BUF_FORMAT. Exporters are free to do what they want with this information. I agree that NumPy would not force you to use it's buffer only as a region of some specific type, but some other object may want to be much more restrictive and only export to consumers who will recognize the data stored for what it is.I think it's up to the exporters to decide whether or not to raise an error when a certain kind of buffer is requested. Basically, every flag corresponds to a different property of the buffer that the consumer is requesting: Py_BUF_SIMPLE --- you are requesting the simplest possible (0x00) Py_BUF_WRITEABLE -- get a writeable buffer (0x01) Py_BUF_READONLY -- get a read-only buffer(0x02) Py_BUF_FORMAT -- get a formatted buffer. (0x04) Py_BUF_SHAPE -- get a buffer with shape information (0x08) Py_BUF_STRIDES -- get a buffer with stride information (and shape) (0x18) Py_BUF_OFFSET -- get a buffer with suboffsets (and strides and shape) (0x38) This is a logical sequence. There is progression. Each flag is a bit that indicates something about how the consumer can use the buffer. In other words, the consumer states what kind of buffer is being requested. The exporter obliges (and can save possibly significant time if the consumer is not requesting the information it must otherwise produce). I originally suggested a small set of flags that expand the set of allowed buffers. Here's a little Venn diagram of buffers to illustrate what I was thinking: http://www.aerojockey.com/temp/venn.png With no flags, the only buffers allowed to be returned are in the All circle but no others. Add Py_BUF_WRITABLE and now you can export writable buffers as well. Add Py_BUF_STRIDED and the strided circle is opened to you, and so on. My recommendation is, any flag should turn on some circle in the Venn diagram (it could be a circle I didn't draw--shaped arrays, for example--but it should be *some* circle). I don't think your Venn diagram is broad enough and it un-necessarily limits the use of flags to communicate between consumer and exporter. We don't have to ram these flags down that point-of-view for them to be productive.If you have a specific alternative proposal, or specific criticisms, then I'm very willing to hear them. I've thought through the flags again, and I'm not sure how I would change them. They make sense to me. Especially in light of past usages of the buffer protocol (where most people request read-or-write buffers i.e. Py_BUF_SIMPLE. I'm also not sure our mental diagrams are both oriented the same. For me, the most restrictive requests are PY_BUF_WRITEABLE | Py_BUF_FORMAT and Py_BUF_READONLY | Py_BUF_FORMAT The most un-restrictive request (the largest circle in my mental Venn diagram) is Py_BUF_OFFSETS followed by Py_BUF_STRIDES followed by Py_BUF_SHAPE adding Py_BUF_FORMATS, Py_BUF_WRITEABLE, or Py_BUF_READONLY serves to restrict any of the other circles Is this dual use of flags what bothers you? (i.e. use of some flags for restricting circles in your Venn diagram that are turned on by other flags? --- you say Py_BUF_OFFSETS | Py_BUF_WRITEABLE to get the intersection of the Py_BUF_OFFSETS largest circle with the WRITEABLE subset?) Such concerns are not convincing to me. Just don't think of the flags in that way. Think of them as turning on members of the bufferinfo structure. Py_BUF_FORMAT The consumer will be using the format string information so make sure thatmember is filled correctly. Is the idea to throw an exception if there's some other data format besides b, and this flag isn't set? It seems superfluous otherwise. The idea is that a consumer may not care about the format and the exporter may want to know that to simplify the interface.In other words the flag is a way for the consumer to communicate that it wants
Re: [Python-Dev] minidom and DOM level 2
Jason Orendorff wrote: I don't suppose you'd be willing to update it for Python 2.5, would you? Can do, but at this point I'm not aware of any work having been done on the issues listed there between the 2.3 and 2.5 releases. The danger is people may be used to the wrong minidom behaviours, given they have been static for so long and are in many cases central to how minidom works. -- And Clover mailto:[EMAIL PROTECTED] http://www.doxdesk.com/ ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] minidom and DOM level 2
On 4/13/07, Andrew Clover [EMAIL PROTECTED] wrote:
Jason Orendorff wrote:
I don't suppose you'd be willing to update it for Python 2.5, would you?
Can do, but at this point I'm not aware of any work having been done on
the issues listed there between the 2.3 and 2.5 releases.
I've been running the DOM test suite against trunk, using your test
harness. It's kind of alarming at first that over 100 tests fail. :)
But many of the failures involve entity references.
An even larger portion involve error cases: we accept things we should
check and reject. For example, doc.createElement('\t') should fail.
These are certainly bugs, and they're easy to fix. I'm working
through them.
The danger is people may be used to the wrong minidom behaviours,
given they have been static for so long and are in many cases central to
how minidom works.
When I get to these, I'll post about it.
-j
___
Python-Dev mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] build problem on windows: unable to find getbuildinfo2.c
On 4/13/07, Martin v. Löwis [EMAIL PROTECTED] wrote: I didn't find getbuildinfo2.c in the source. Can some one tell me if I am missing some thing here? Are there any additional steps need to follow on windows? It's a generated file. Search all build description files for that file name to find out how it is generated, and then research why generating it fails on your machine. Thanks. I found that make_buildinfo project failed with some unresolved symbol errors causing the missing getbuildinfo2.c. Once I fixed it, python got built just fine. ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3118: Extended buffer protocol (new version)
On 4/13/07, Travis Oliphant [EMAIL PROTECTED] wrote: int PyObject_GetContiguous(PyObject *obj, void **buf, Py_ssize_t *len, int fortran) Return a contiguous chunk of memory representing the buffer. If a copy is made then return 1. If no copy was needed return 0. 8) If a copy was made, What should consumers call to free memory? You are right. We need a free function. I think now the memory perhaps should be allocated with PyMem_New and deallocated with PyMem_Free. Additionally, the return should perhaps indicate success or failure, and a new argument should be passed in order to know if a copy was made, something like int PyObject_GetContiguous(PyObject *obj, void **buf, Py_ssize_t *len, int *copy, char layout) -- Lisandro Dalcín --- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_pty.py hangs in verbose mode on Mac OS X?
On Fri, 13 Apr 2007 10:32:28 -0400, Barry Warsaw [EMAIL PROTECTED] wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I've been getting some test failures in Python 2.5 svn head on Mac OS X 10.4.9 which I'm not getting on Linux (Ubuntu feisty beta). test_sqlite and test_zipimport both fail, however, when run in verbose mode (e.g. ./python.exe Lib/test/test_sqlite.py) both pass. But that's not exactly why I'm writing this email wink. In the course of trying to debug this, I ran the following on my Mac: make TESTOPTS=-v test This runs the entire test suite in verbose mode, and you do get a lot of output. However the test suite hangs on test_pty.py. In fact, if you run that test alone: ./python.exe Lib/test/test_pty.py it too hangs for me. The reason is that in verbose mode, debug() actually prints stuff to stdout and on the Mac, when the child of the pty.fork() writes to its stdout, it blocks and so the parent's waitpid () never returns. This doesn't happen on Linux though; the child's stdout prints don't block, it exits, and the parent continues after the waitpid(). Here's a very simple program that reproduces the problem: - -snip snip- import os, pty, sys pid, fd = pty.fork() print sys.stderr, pid, fd if pid: os.waitpid(pid, 0) else: os._exit(0) - -snip snip- stderr, stdout, doesn't matter. This hangs on the Mac but completes successfully on Linux. Of course, in neither case do you see the child's output. I don't know if this is caused by a bug in the Mac's pty implementation or something we're doing wrong on that platform. I played around with several modifications to pty.fork() on the Mac, including letting it drop down to the openpty()/os.fork() code, even adding an explicit ioctl(slave_fd, TIOCSCTTY) call which Stevens chapter 19 recommends for 4.3+BSD. I can't get it to not block. What about reading from the child in the parent before calling waitpid? Barring a fix to pty.fork() (or possibly os.forkpty()) for the Mac, then I would like to at least make test_pty.py not block when run in verbose mode. A very simple hack would add something like this to the if pid == pty.CHILD stanza: def debug(msg): pass, possibly protected by a if verbose:. A less icky hack would be to read the output from the master_fd in the parent, though you have to be careful with that on Linux else the read can throw an input/output error. Disabling debug output is band-aid yes, and any application on the Mac like the above snippet will still fail. If anybody has any suggestions, I'm all ears, but I've reached the limit of my pty-fu. I don't think this is an OS X PTY bug. Writing to a blocking file descriptor can block. Programs that do this need to account for the possibility. Jean-Paul ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_pty.py hangs in verbose mode on Mac OS X?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Apr 13, 2007, at 10:57 AM, Jean-Paul Calderone wrote: I don't know if this is caused by a bug in the Mac's pty implementation or something we're doing wrong on that platform. I played around with several modifications to pty.fork() on the Mac, including letting it drop down to the openpty()/os.fork() code, even adding an explicit ioctl(slave_fd, TIOCSCTTY) call which Stevens chapter 19 recommends for 4.3+BSD. I can't get it to not block. What about reading from the child in the parent before calling waitpid? Yep, this is what I suggested below. Porting the same change over to Linux produced an OSError, but that's probably just because I wasn't as careful as I should have been late last night. Barring a fix to pty.fork() (or possibly os.forkpty()) for the Mac, then I would like to at least make test_pty.py not block when run in verbose mode. A very simple hack would add something like this to the if pid == pty.CHILD stanza: def debug(msg): pass, possibly protected by a if verbose:. A less icky hack would be to read the output from the master_fd in the parent, though you have to be careful with that on Linux else the read can throw an input/output error. Disabling debug output is band-aid yes, and any application on the Mac like the above snippet will still fail. If anybody has any suggestions, I'm all ears, but I've reached the limit of my pty-fu. I don't think this is an OS X PTY bug. Writing to a blocking file descriptor can block. Programs that do this need to account for the possibility. Why doesn't it block on Linux then? - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRh+banEjvBPtnXfVAQJxFQP+LRAdGVHuqquvGNT9fncGeJFuCjGm+dsS VnE8cirvfoSEdJ0nZ11wHMoMH2vtbXyfjLJJ4G/sROMmHKWJ2t5ieNIxmVutiiLa OPyls2bzuXL8IoyYh+c8tKRyBd76O5GN2EZABxGm2Tie5nt72ezVSEfiDovc6qEu 4lYERTD1YKA= =fZaZ -END PGP SIGNATURE- ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_pty.py hangs in verbose mode on Mac OS X?
On Fri, 13 Apr 2007 11:02:01 -0400, Barry Warsaw [EMAIL PROTECTED] wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Apr 13, 2007, at 10:57 AM, Jean-Paul Calderone wrote: I don't know if this is caused by a bug in the Mac's pty implementation or something we're doing wrong on that platform. I played around with several modifications to pty.fork() on the Mac, including letting it drop down to the openpty()/os.fork() code, even adding an explicit ioctl(slave_fd, TIOCSCTTY) call which Stevens chapter 19 recommends for 4.3+BSD. I can't get it to not block. What about reading from the child in the parent before calling waitpid? Yep, this is what I suggested below. Porting the same change over to Linux produced an OSError, but that's probably just because I wasn't as careful as I should have been late last night. Barring a fix to pty.fork() (or possibly os.forkpty()) for the Mac, then I would like to at least make test_pty.py not block when run in verbose mode. A very simple hack would add something like this to the if pid == pty.CHILD stanza: def debug(msg): pass, possibly protected by a if verbose:. A less icky hack would be to read the output from the master_fd in the parent, though you have to be careful with that on Linux else the read can throw an input/output error. Disabling debug output is band-aid yes, and any application on the Mac like the above snippet will still fail. If anybody has any suggestions, I'm all ears, but I've reached the limit of my pty-fu. I don't think this is an OS X PTY bug. Writing to a blocking file descriptor can block. Programs that do this need to account for the possibility. Why doesn't it block on Linux then? Likely differing buffering behavior. Prior to Linux 2.6, the pipe implementation allowed only a single buffer (that is, the bytes from a single write call) in a pipe at a time, and blocked subsequent writes until that buffer was read. Recently this has changed to allow multiple buffers up to 4k total length, so multiple short writes won't block anymore. OS X may have some other buffering behavior which is causing writes to block where they don't on Linux. All these details are left to the platform, and there are a variety of behaviors which can be considered valid. Of course, I don't actually /know/ the cause of the problem here, but this explanation seems plausible to me, and I'd investigate it before looking for platform-specific pty bugs (although OS X is a good platform on which to go looking for those ;). Jean-Paul ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Pydoc Rewrite Discussion at doc-sig list.
If anyone is interested in participating in discussing the details of the PyDoc rewrite/refactoring I've been working on, a discussion is being started on the doc-sig list. [EMAIL PROTECTED] The goal of this discussion will be to get it to a final finished form so a patch can be submitted and a final discussion can take place on the python-dev list at a later date. Thanks and Regards, Ron Adam ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_pty.py hangs in verbose mode on Mac OS X?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Apr 13, 2007, at 11:07 AM, Jean-Paul Calderone wrote: Likely differing buffering behavior. Prior to Linux 2.6, the pipe implementation allowed only a single buffer (that is, the bytes from a single write call) in a pipe at a time, and blocked subsequent writes until that buffer was read. Recently this has changed to allow multiple buffers up to 4k total length, so multiple short writes won't block anymore. OS X may have some other buffering behavior which is causing writes to block where they don't on Linux. All these details are left to the platform, and there are a variety of behaviors which can be considered valid. Of course, I don't actually /know/ the cause of the problem here, but this explanation seems plausible to me, and I'd investigate it before looking for platform-specific pty bugs (although OS X is a good platform on which to go looking for those ;). Seems plausible to me too. If I change the child writes to 4000 bytes, Linux w/2.6 kernel will block too. I'm going to change the test_pty.py code to read from the master_fd. You have to wrap the read in a try/except to catch the OSError input/ output error you'll get on Linux when you read past the end of the buffer and the child process has already exited. Thanks, - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRh+gO3EjvBPtnXfVAQJkQAP9E2Jp/EeF3ChIeNZEPVMDeG4Kd+PVslSs blNFO2oO6eO8Yn9X3hlnLwCe3W6+89bS9u7MyN1C2CCZTBz6N1QAkfDaAnxuhbNC 3hfvytMguTY0v3RiZhEVY+y/h2PfbWe6fJGXeBIfcth1HWhP0KWIgAoDA+odDm1B vZSvcxBRN1Y= =fDrh -END PGP SIGNATURE- ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Changes to decimal.py
[Raymond Hettinger] ... Likewise, consider soliciting Tim's input on how to implement the ln() operation. That one will be tricky to get done efficiently and correctly. One low-effort approach is to use a general root-finding algorithm and build ln(x) on top of exp() via (numerically) solving the equation exp(ln(x)) == x for ln(x). That appears to be what Don Peterson did in his implementation of transcendental functions for Decimal: http://cheeseshop.python.org/pypi/decimalfuncs/1.4 In a bit of disguised form, that appears to be what Brian Beck and Christopher Hesse also did: http://cheeseshop.python.org/pypi/dmath/0.9 The former is GPL-licensed and the latter MIT, so the latter would be easier to start from for core (re)distribution. However, the IBM spec requires 1 ULP worst-case error, and that may be unreasonably hard to meet with a root-finding approach. If this can wait a couple months, I'd be happy to own it. A possible saving grace for ln() is that while the mathematical function is one-to-one, in any fixed precision it's necessarily many-to-one (e.g., log10 of the representable numbers between 10 and 1e100 must be a representable number between 1 and 100, and there are a lot more of the former than of the latter -- many distinct representable numbers must map to the same representable log). ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] ok to just checkin minor obvious fixes?
Is it generally accepted to just checkin obviously non-controversial fixes (*) (as long as there is no code freeze), or is it still preferred that I go through adding a patch to the SF tracker and getting review? Trent (*) In my case a tweak to the old VC6 Windows build files to ensure a clean build works (_ctypes project must depend on pythoncore proj). -- Trent Mick trentm at activestate.com ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] my 2.5 checkins
Hello all. I made two checkins to the 25 maintainance branch before Martin kindly pointed out to me that it is frozen. These are quite simple fixes to real crashes I have experienced. The fix in frameobject.c will be necessary if you work with opcodes 128, which we routinely do at CCP J. Security through opcode randomization Anyway, just let me know if you like me to roll them back. Cheers, Kristján ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3118: Extended buffer protocol (new version)
Travis Oliphant wrote: Py_BUF_SIMPLE --- you are requesting the simplest possible (0x00) Py_BUF_WRITEABLE -- get a writeable buffer (0x01) Py_BUF_READONLY -- get a read-only buffer(0x02) I don't see how these three form a progression. From the other things you say, it appears you mean Py_BUF_SIMPLE to mean both readable and writeable. But then Py_BUF_READONLY is turning off a capability (being able to write to the buffer) rather than turning one on. Seems to me the simplest possible buffer is a read-only one (assuming we don't want to allow for write-only buffers -- or do we?), in which case a more logical arrangement to my mind would be Py_BUF_READONLY = 0x00 # simplest possible PY_BUF_READWRITE = 0x01 If we do want write-only buffers, then there isn't a single simplest possible buffer, except for one that's neither readable nor writable, which doesn't seem very useful. So we would have Py_BUF_READABLE = 0x01 Py_BUF_WRITABLE = 0x02 Py_BUF_READWRITE = 0x03 -- Greg ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
