Re: [Python-Dev] idea for data-type (data-format) PEP
Alexander Belopolsky schrieb: Multi-segment buffers are only dead because standard library modules do not support them. That, in turn, is because nobody has contributed code to make that work. My guess is that people either don't need it, or find it too difficult to implement. In any case, it is an important point that such a specification is likely dead if the standard library doesn't support it throughout, from start. So for this PEP, the same criterion likely applies: it's not sufficient to specify an interface, one also has to specify (and then implement) how that affects modules and types of the standard library. I often work with text data that is represented as an array of strings. I would love to implement a multi-segment buffer interface on top of that data and be able to do a full text regular expression search without having to concatenate into one big string, but python's re module would not take a multi-segment buffer. If you are curious, try adding such a feature to re some time. I expect that implementing it would be quite involved. I wonder what Fredrik Lundh thinks about providing such a feature. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Feature Request: Py_NewInterpreter to create separate GIL (branch)
Robert schrieb: Would it be a possibilty in next Python to have the lock separate for each Interpreter instance. Thus: have *interpreter_lock separate in each PyThreadState instance and only threads of same Interpreter have same GIL? Separation between Interpreters seems to be enough. The Interpreter runs mainly on the stack. Possibly only very few global C-level resources would require individual extra locks. Notice that at least the following objects are shared between interpreters, as they are singletons: - None, True, False, (), , u - strings of length 1, Unicode strings of length 1 with ord 256 - integers between -5 and 256 How do you deal with the reference counters of these objects? Also, type objects (in particular exception types) are shared between interpreters. These are mutable objects, so you have actually dictionaries shared between interpreters. How would you deal with these? Also, the current thread state is a global variable, currently (_PyThreadState_Current). How would you provide access to the current thread state if there are multiple simultaneous threads? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Importing .pyc in -O mode and vice versa
Patch #1346572 proposes to also search for .pyc when OptimizeFlag is set, and for .pyo when it is not set. The author argues this is for consistency, as the zipimporter already does that. This reasoning is somewhat flawed, of course: to achieve consistency, one could also change the zipimporter instead. However, I find the proposed behaviour reasonable: Python already automatically imports the .pyc file if .py is not given and vice versa. So why not look for .pyo if the .pyc file is not present? What do you think? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Path object design
On 11/3/06, Steve Holden [EMAIL PROTECTED] wrote: Having said this, Andrew *did* demonstrate quite convincingly that the current urljoin has some fairly egregious directory traversal glitches. Is it really right to punt obvious gotchas like urlparse.urljoin(http://blah.com/a/b/c;, ../../../../) 'http://blah.com/../../' Ah, but how do you know when that's wrong? At least under ftp:// your root is often a mid-level directory until you change up out of it. http:// will tend to treat the targets as roots, but I don't know that there's any requirement for a /.. to be meaningless (even if it often is). -- Michael Urman http://www.tortall.net/../mu/blog ;) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
On Sat, Nov 04, 2006 at 04:47:37PM +0100, Martin v. L?wis wrote: Patch #1346572 proposes to also search for .pyc when OptimizeFlag is set, and for .pyo when it is not set. The author argues this is for consistency, as the zipimporter already does that. This reasoning is somewhat flawed, of course: to achieve consistency, one could also change the zipimporter instead. However, I find the proposed behaviour reasonable: Python already automatically imports the .pyc file if .py is not given and vice versa. So why not look for .pyo if the .pyc file is not present? What do you think? +1 from me. Oleg. -- Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] idea for data-type (data-format) PEP
On Nov 4, 2006, at 3:15 AM, Martin v. Löwis wrote: Alexander Belopolsky schrieb: Multi-segment buffers are only dead because standard library modules do not support them. That, in turn, is because nobody has contributed code to make that work. My guess is that people either don't need it, or find it too difficult to implement. Last time I tried to contribute code related to buffer protocol, it was rejected with little discussion http://sourceforge.net/tracker/index.php? func=detailaid=1539381group_id=5470atid=305470 that patch implemented two features: enabled creation of read-write buffer objects and added readinto method to StringIO. The resolution was: The file object's readinto method is not meant for public use, so adding the method to StringIO is not a good idea. The read-write buffer part was not discussed, but I guess the resolution would be that buffer objects are deprecated, so adding features to them is not a good idea. If you are curious, try adding such a feature to re some time. I expect that implementing it would be quite involved. I wonder what Fredrik Lundh thinks about providing such a feature. I would certainly invest some time into that if that feature had a chance of being accepted. At the moment I feel that anything related to buffers or buffer protocol is met with strong opposition. I think the opposition is mostly fueled by the belief that buffer objects are unsafe and buffer protocol is deprecated. None of these premises is correct AFAIK. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Path object design
Michael Urman wrote: On 11/3/06, Steve Holden [EMAIL PROTECTED] wrote: Having said this, Andrew *did* demonstrate quite convincingly that the current urljoin has some fairly egregious directory traversal glitches. Is it really right to punt obvious gotchas like urlparse.urljoin(http://blah.com/a/b/c;, ../../../../) 'http://blah.com/../../' Ah, but how do you know when that's wrong? At least under ftp:// your root is often a mid-level directory until you change up out of it. http:// will tend to treat the targets as roots, but I don't know that there's any requirement for a /.. to be meaningless (even if it often is). I'm darned if I know. I simply know that it isn't right for http resources. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Fredrik Lundh schrieb: However, I find the proposed behaviour reasonable: Python already automatically imports the .pyc file if .py is not given and vice versa. So why not look for .pyo if the .pyc file is not present? well, from a performance perspective, it would be nice if Python looked for *fewer* things, not more things. That's true. (wouldn't transparent import of PYO files mean that you end up with a program where some assertions apply, and others don't? could be con- fusing...) That's also true, however, it might still be better to do that instead of raising an ImportError. I'm not sure whether a scenario were you have only .pyo files for some modules and only .pyc files for others is really likely, though, and the performance hit of another system call doesn't sound attractive. So I guess that zipimport should stop importing .pyo files if OptimizeFlag is false, then? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Path object design
Steve Holden wrote: Ah, but how do you know when that's wrong? At least under ftp:// your root is often a mid-level directory until you change up out of it. http:// will tend to treat the targets as roots, but I don't know that there's any requirement for a /.. to be meaningless (even if it often is). I'm darned if I know. I simply know that it isn't right for http resources. the URI specification disagrees; an URI that starts with ../ is per- fectly legal, and the specification explicitly states how it should be interpreted. (it's important to realize that urijoin produces equivalent URI:s, not file names) /F ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of pairing_heap.py?
Paul Chiusano schrieb: To support this, the insert method needs to return a reference to an object which I can then pass to adjust_key() and delete() methods. It's extremely difficult to have this functionality with array-based heaps because the index of an item in the array changes as items are inserted and removed. I see. Okay, I'll do that. What needs to be done to move the project along and possibly get a pairing heap incorporated into a future version of python? As a starting point, I think the implementation should get packaged as an independent library, and be listed in the Cheeseshop for a few years. If then there's wide interest in including it into Python, it should be reconsidered. At that point, the then-authors of the package will have to sign a contributor form. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Hi, I'm the author of this patch and we are already using it in Python port for Maemo platform. We are using .pyo modules mainly to remove docstrings from the modules. We've discussed about this patch here[1] before. Now, I agree that the zipimport behaviour is incorrect but I don't have other option to remove docstrings of a .pyc file. I'm planning to send a patch that adds a --remove-docs to the Python interpreter to replace the -OO option that create only .pyo files. [1] http://mail.python.org/pipermail/python-dev/2005-November/057959.html On 11/4/06, Martin v. Löwis [EMAIL PROTECTED] wrote: Fredrik Lundh schrieb: However, I find the proposed behaviour reasonable: Python already automatically imports the .pyc file if .py is not given and vice versa. So why not look for .pyo if the .pyc file is not present? well, from a performance perspective, it would be nice if Python looked for *fewer* things, not more things. That's true. [cut] -- Osvaldo Santana Neto (aCiDBaSe) http://www.pythonologia.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
On 11/4/06, Martin v. Löwis [EMAIL PROTECTED] wrote: Fredrik Lundh schrieb: However, I find the proposed behaviour reasonable: Python already automatically imports the .pyc file if .py is not given and vice versa. So why not look for .pyo if the .pyc file is not present? well, from a performance perspective, it would be nice if Python looked for *fewer* things, not more things.That's true. (wouldn't transparent import of PYO files mean that you end up with a program where some assertions apply, and others don't?could be con- fusing...)That's also true, however, it might still be better to do that insteadof raising an ImportError.I'm not sure whether a scenario were you have only .pyo files for some modules and only .pyc files for others is really likely, though,and the performance hit of another system call doesn't sound attractive.So I guess that zipimport should stop importing .pyo files if OptimizeFlag is false, then?Yes, I think it should. When I get around to rewriting zipimport for my import rewrite it will do this by default.-Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
On 11/4/06, Osvaldo Santana [EMAIL PROTECTED] wrote: Hi,I'm the author of this patch and we are already using it in Pythonport for Maemo platform.We are using .pyo modules mainly to remove docstrings from themodules. We've discussed about this patch here[1] before. Now, I agree that the zipimport behaviour is incorrect but I don'thave other option to remove docstrings of a .pyc file.I'm planning to send a patch that adds a --remove-docs to the Python interpreter to replace the -OO option that create only .pyo files.[1] http://mail.python.org/pipermail/python-dev/2005-November/057959.html The other option is to do away with .pyo files: http://www.python.org/dev/summary/2005-11-01_2005-11-15/#importing-pyc-and-pyo-files Guido has said he wouldn't mind it, but then .pyc files need to grow a field or so to be able to store what optimizations were used. While this would lead to more bytecode regeneration, it would help deal with this case and allow for more optimizations on the bytecode. -BrettOn 11/4/06, Martin v. Löwis [EMAIL PROTECTED] wrote: Fredrik Lundh schrieb: However, I find the proposed behaviour reasonable: Python already automatically imports the .pyc file if .py is not given and vice versa. So why not look for .pyo if the .pyc file is not present? well, from a performance perspective, it would be nice if Python looked for *fewer* things, not more things. That's true.[cut]--Osvaldo Santana Neto (aCiDBaSe)http://www.pythonologia.org___Python-Dev mailing list Python-Dev@python.orghttp://mail.python.org/mailman/listinfo/python-devUnsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of pairing_heap.py?
Martin v. Löwis [EMAIL PROTECTED] wrote: Paul Chiusano schrieb: To support this, the insert method needs to return a reference to an object which I can then pass to adjust_key() and delete() methods. It's extremely difficult to have this functionality with array-based heaps because the index of an item in the array changes as items are inserted and removed. I see. It is not required. If you are careful, you can implement a pairing heap with a structure combining a dictionary and list. It requires that all values be unique and hashable, but it is possible (I developed one for a commercial project). If other people find the need for it, I could rewrite it (can't release the closed source). It would use far less memory than the pairing heap implementation provided in the sandbox, and could be converted to C if desired and/or required. On the other hand, I've found the pure Python version to be fast enough for most things I've needed it for. - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Fredrik Lundh wrote: well, from a performance perspective, it would be nice if Python looked for *fewer* things, not more things. Instead of searching for things by doing a stat call for each possible file name, would it perhaps be faster to read the contents of all the directories along sys.path into memory and then go searching through that? -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
On Sun, 05 Nov 2006 14:21:34 +1300, Greg Ewing [EMAIL PROTECTED] wrote: Fredrik Lundh wrote: well, from a performance perspective, it would be nice if Python looked for *fewer* things, not more things. Instead of searching for things by doing a stat call for each possible file name, would it perhaps be faster to read the contents of all the directories along sys.path into memory and then go searching through that? Bad for large directories. There's a cross-over at some number of entries. Maybe Python should have a runtime-tuned heuristic for selecting a filesystem traversal mechanism. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Greg Ewing schrieb: Fredrik Lundh wrote: well, from a performance perspective, it would be nice if Python looked for *fewer* things, not more things. Instead of searching for things by doing a stat call for each possible file name, would it perhaps be faster to read the contents of all the directories along sys.path into memory and then go searching through that? That should never be better: the system will cache the directory blocks, also, and it will do a better job than Python will. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
On 11/4/06, Jean-Paul Calderone [EMAIL PROTECTED] wrote: On Sun, 05 Nov 2006 14:21:34 +1300, Greg Ewing [EMAIL PROTECTED] wrote:Fredrik Lundh wrote: well, from a performance perspective, it would be nice if Python looked for *fewer* things, not more things.Instead of searching for things by doing a stat callfor each possible file name, would it perhaps befaster to read the contents of all the directories along sys.path into memory and then go searchingthrough that?Bad for large directories.There's a cross-over at some numberof entries.Maybe Python should have a runtime-tuned heuristicfor selecting a filesystem traversal mechanism. Hopefully my import rewrite is flexible enough that people will be able to plug in their own importer/loader for the filesystem so that they can tune how things like this are handled (e.g., caching what files are in a directory, skipping bytecode files, etc.). -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
