Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Scott David Daniels
A.M. Kuchling wrote: With zipfile, you could at least access the .fp attribute to sync it (though is the .fp documented as part of the interface?). For this one, I'd like to add the sync as a method (so that Zip-inside- Zip is eventually possible). In fact, a sync on an exposed writable

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Martin v. Löwis
We already have os.fsync() and os.fdatasync(). Should the sync() (and datasync()?) method be added as an object-oriented convenience? It's more than an object oriented convenience. fsync() takes a file descriptor as argument. Therefore I assume fsync() only syncs the data to disk that was

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Joachim König
Guido van Rossum wrote: On Tue, Mar 10, 2009 at 1:11 PM, Christian Heimes li...@cheimes.de wrote: [...] https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/54. [...] If I understand the post properly, it's up to the app to call fsync(), and it's only necessary

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Neil Hodgson
Antoine Pitrou: How about shutil.copystat()? shutil.copystat does not copy over the owner, group or ACLs. Modeling a copymetadata method on copystat would provide an easy to understand API and should be implementable on Windows and POSIX. Reading the OS X documentation shows a set of

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Hrvoje Niksic
Joachim König wrote: To me, the flaw seem to be in the close() call (of the operating system). I'd expect the data to be in a persistent state once the close() returns. I wouldn't, because that would mean that every cp -r would effectively do an fsync() for each individual file it copies,

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Antoine Pitrou
Neil Hodgson nyamatongwe at gmail.com writes: shutil.copystat does not copy over the owner, group or ACLs. It depends on what you call ACLs. It does copy the chmod permission bits. As for owner and group, I think there is a very good reason that it doesn't copy them: under Linux, only root

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Oleg Broytmann
On Wed, Mar 11, 2009 at 11:43:33AM +, Antoine Pitrou wrote: As for owner and group, I think there is a very good reason that it doesn't copy them: under Linux, only root can change these properties. Only root can change file ownership - and yes, there are scripts that run with root

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Antoine Pitrou
Christian Heimes lists at cheimes.de writes: It's more than an object oriented convenience. fsync() takes a file descriptor as argument. Therefore I assume fsync() only syncs the data to disk that was written to the file descriptor. Ok, I agree that a .sync() method makes sense.

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Antoine Pitrou
Oleg Broytmann phd at phd.pp.ru writes: Only root can change file ownership - and yes, there are scripts that run with root privileges, so why not copy? Because the new function would then be useless for non-root scripts, and encouraging people to run their scripts as root would be rather

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Antoine Pitrou
Oleg Broytmann phd at phd.pp.ru writes: That's easy to fix - only copy ownership if the effective user id == 0. But errors should not pass silently. If the user intended the function to copy ownership information and the function fails to do so, it should raise an exception. Having

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Christian Heimes
Guido van Rossum wrote: Let's not think too Unix-specific. If we add such an API it should do something on Windows too -- the app shouldn't have to test for the presence of the API. (And thus the API probably shouldn't be called fsync.) In my initial proposal one and a half hour earlier I

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Hrvoje Niksic
Christian Heimes wrote: Guido van Rossum wrote: Let's not think too Unix-specific. If we add such an API it should do something on Windows too -- the app shouldn't have to test for the presence of the API. (And thus the API probably shouldn't be called fsync.) In my initial proposal one and a

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Antoine Pitrou
Christian Heimes lists at cheimes.de writes: In my initial proposal one and a half hour earlier I suggested 'sync()' as the name of the method and 'synced' as the name of the flag that forces a fsync() call during the close operation. I think your synced flag is too vague. Some applications

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Antoine Pitrou
After Hrvoje's message, let me rephrase my suggestion. Let's instead allow: open(..., sync_on=close) open(..., sync_on=flush) with a default of None meaning no implicit syncs. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Antoine Pitrou
Christian Heimes lists at cheimes.de writes: And sync_on=flush implies sync_on=close? close() implies flush(), so by construction yes. Your suggestion sounds like the right way to me! I'm glad I brought something constructive to the discussion :-))

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Scott Dial
Aahz wrote: On Wed, Mar 11, 2009, Antoine Pitrou wrote: After Hrvoje's message, let me rephrase my suggestion. Let's instead allow: open(..., sync_on=close) open(..., sync_on=flush) with a default of None meaning no implicit syncs. That looks good, though I'd prefer using named

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Aahz
On Wed, Mar 11, 2009, Scott Dial wrote: Aahz wrote: On Wed, Mar 11, 2009, Antoine Pitrou wrote: After Hrvoje's message, let me rephrase my suggestion. Let's instead allow: open(..., sync_on=close) open(..., sync_on=flush) with a default of None meaning no implicit syncs. That looks

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Eric Smith
Antoine Pitrou wrote: I think your synced flag is too vague. Some applications may need the file to be synced on close(), but some others may need it to be synced at regular intervals, or after each write(), etc. Why wouldn't sync just be an optional argument to close(), at least for the

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Eric Smith
Antoine Pitrou wrote: Eric Smith eric at trueblade.com writes: Why wouldn't sync just be an optional argument to close(), at least for the sync_on_close case? It wouldn't work with the with statement. Well, that is a good reason, then! ___

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Martin v. Löwis
Maybe it would make more sense for synced to force fsync() on each flush, not only on close. I'm not sure how useful it is, but that's what synced would imply to me. That should be implement by passing O_SYNC on open, rather than explicitly calling fsync. Regards, Martin

[Python-Dev] Can I modify string.Formatter._vformat?

2009-03-11 Thread Eric Smith
I'm implementing support for auto-numbering of str.format() fields (see http://bugs.python.org/issue5237). I'm reasonably sure that when I'm done modifying the C implementation I'll need to change the signatures of string.Formatter._vformat, str._formatter_parser, and/or

Re: [Python-Dev] Can I modify string.Formatter._vformat?

2009-03-11 Thread Benjamin Peterson
2009/3/11 Eric Smith e...@trueblade.com: I'm implementing support for auto-numbering of str.format() fields (see http://bugs.python.org/issue5237). I'm reasonably sure that when I'm done modifying the C implementation I'll need to change the signatures of string.Formatter._vformat,

Re: [Python-Dev] Can I modify string.Formatter._vformat?

2009-03-11 Thread Brett Cannon
On Wed, Mar 11, 2009 at 13:20, Benjamin Peterson benja...@python.orgwrote: 2009/3/11 Eric Smith e...@trueblade.com: I'm implementing support for auto-numbering of str.format() fields (see http://bugs.python.org/issue5237). I'm reasonably sure that when I'm done modifying the C

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Neil Hodgson
Antoine Pitrou: It depends on what you call ACLs. It does copy the chmod permission bits. Access Control Lists are fine grained permissions. Perhaps you want to allow Sam to read a file and for Ted to both read and write it. These permissions should not need to be reset every time you

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Greg Ewing
Barry Warsaw wrote: Of course, a careful *nix application can ensure that the file owners and mod bits are set the way it needs them to be set. A convenience function might be useful though. A specialised function would also provide a place for dealing with platform-specific extensions,

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Cameron Simpson
On 11Mar2009 10:09, Joachim K?nig h...@online.de wrote: Guido van Rossum wrote: On Tue, Mar 10, 2009 at 1:11 PM, Christian Heimes li...@cheimes.de wrote: [...] https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/54. [...] If I understand the post properly, it's up to

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Steven D'Aprano
On Thu, 12 Mar 2009 01:21:25 am Antoine Pitrou wrote: Christian Heimes lists at cheimes.de writes: In my initial proposal one and a half hour earlier I suggested 'sync()' as the name of the method and 'synced' as the name of the flag that forces a fsync() call during the close operation.

Re: [Python-Dev] [Python-ideas] Adding a test discovery into Python

2009-03-11 Thread Guilherme Polo
On Wed, Mar 11, 2009 at 7:37 PM, Raymond Hettinger pyt...@rcn.com wrote: [Christian Heimes] I'm +1 for a simple (!) test discovery system. I'm emphasizing on simple because there are enough frameworks for elaborate unit testing. Test discovery is not the interesting part of the problem.

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Greg Ewing
Lie Ryan wrote: I actually prefer strings. Just like 'w' or 'r' in open(). Or why not add f c as modes? open('file.txt', 'wf') I like this, because it doesn't expand the signature that file-like objects need to support. If you're wrapping another file object you just need to pass on the

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Greg Ewing
Martin v. Löwis wrote: That should be implement by passing O_SYNC on open, rather than explicitly calling fsync. On platforms which have it (MacOSX doesn't seem to, according to the man page). This is another good reason to put these things in the mode string. -- Greg

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Paul Moore
2009/3/11 Greg Ewing greg.ew...@canterbury.ac.nz: Lie Ryan wrote: I actually prefer strings. Just like 'w' or 'r' in open(). Or why not add f c as modes? open('file.txt', 'wf') I like this, because it doesn't expand the signature that file-like objects need to support. If you're wrapping

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Antoine Pitrou
Greg Ewing greg.ewing at canterbury.ac.nz writes: I like this, because it doesn't expand the signature that file-like objects need to support. If you're wrapping another file object you just need to pass on the mode string and it will all work. What do you mean? open() doesn't allow you to

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Greg Ewing
Antoine Pitrou wrote: What do you mean? open() doesn't allow you to wrap other file objects. I'm talking about things like GzipFile that take a filename and mode, open the file and then wrap the file object. -- Greg ___ Python-Dev mailing list

[Python-Dev] Formatting mini-language suggestion

2009-03-11 Thread Raymond Hettinger
The current formatting mini-language provisions left/right/center alignment, prefixes for 0b 0x 0o, and rules on when to show the plus-sign. I think it would be far more useful to provision a simple way of specifying a thousands separator. Financial users in particular find the locale approach

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Nick Coghlan
Greg Ewing wrote: Antoine Pitrou wrote: What do you mean? open() doesn't allow you to wrap other file objects. I'm talking about things like GzipFile that take a filename and mode, open the file and then wrap the file object. The tempfile module would be another example. For that

Re: [Python-Dev] Formatting mini-language suggestion

2009-03-11 Thread Antoine Pitrou
Raymond Hettinger python at rcn.com writes: Financial users in particular find the locale approach to be frustrating and non-obvious. Putting in a thousands separator is a common task for output destined to be read by non-programmers. Please note that for it to be useful in all parts of

Re: [Python-Dev] Formatting mini-language suggestion

2009-03-11 Thread Nick Coghlan
Raymond Hettinger wrote: The current formatting mini-language provisions left/right/center alignment, prefixes for 0b 0x 0o, and rules on when to show the plus-sign. I think it would be far more useful to provision a simple way of specifying a thousands separator. Financial users in

Re: [Python-Dev] Formatting mini-language suggestion

2009-03-11 Thread James Y Knight
On Mar 11, 2009, at 9:06 PM, Nick Coghlan wrote: Raymond Hettinger wrote: The current formatting mini-language provisions left/right/center alignment, prefixes for 0b 0x 0o, and rules on when to show the plus-sign. I think it would be far more useful to provision a simple way of specifying a

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Antoine Pitrou
Nick Coghlan ncoghlan at gmail.com writes: The tempfile module would be another example. Do you really need your temporary files to survive system crashes? ;) For that reason, I think Steven's idea of a filetools module which provided context managers and the like that wrapped *existing*

Re: [Python-Dev] [Python-ideas] Ext4 data loss

2009-03-11 Thread zooko
Would there be interest in a filetools module? Replies and discussion to python-ideas please. I've been using and maintaining a few filesystem hacks for, let's see, almost nine years now: http://allmydata.org/trac/pyutil/browser/pyutil/pyutil/fileutil.py (The first version of that was

Re: [Python-Dev] Formatting mini-language suggestion

2009-03-11 Thread Raymond Hettinger
[James Y Knight] You might be interested to know that in India, the commas don't come every 3 digits. In india, they come every two digits, after the first three. Thus one billion = 1,00,00,00,000. How are you gonna represent *that* in a formatting mini-language? :) It is not the goal to

Re: [Python-Dev] Formatting mini-language suggestion

2009-03-11 Thread Ben Finney
James Y Knight f...@fuhm.net writes: You might be interested to know that in India, the commas don't come every 3 digits. In india, they come every two digits, after the first three. Thus one billion = 1,00,00,00,000. How are you gonna represent *that* in a formatting mini-language? :)

Re: [Python-Dev] Formatting mini-language suggestion

2009-03-11 Thread Guido van Rossum
On Wed, Mar 11, 2009 at 6:01 PM, Antoine Pitrou solip...@pitrou.net wrote: Raymond Hettinger python at rcn.com writes: Financial users in particular find the locale approach to be frustrating and non-obvious.  Putting in a thousands separator is a common task for output destined to be read

Re: [Python-Dev] Ext4 data loss

2009-03-11 Thread Nick Coghlan
Antoine Pitrou wrote: Nick Coghlan ncoghlan at gmail.com writes: The tempfile module would be another example. Do you really need your temporary files to survive system crashes? ;) No, but they need to provide the full file API. If we add a sync() method to file objects, that becomes part of

[Python-Dev] Approaches to argument type-checking

2009-03-11 Thread Tennessee Leeuwenburg
Hi all, I am currently looking at issue 5236. This issue regards the exception raised when a bytes string is passed into time.strptime. In addition to the specific question I have regarding this issue, I wasn't sure if this was something for python-dev or for the issue comment. However, it does

Re: [Python-Dev] Approaches to argument type-checking

2009-03-11 Thread Benjamin Peterson
2009/3/11 Tennessee Leeuwenburg tleeuwenb...@gmail.com: Is there a general strategy used in Python development which I should be aware of? i.e. is it customary to type-check every argument of an interface method? Or is it customary not to perform type checking up-front and simply allow the

Re: [Python-Dev] Formatting mini-language suggestion

2009-03-11 Thread Nick Coghlan
Raymond Hettinger wrote: [James Y Knight] You might be interested to know that in India, the commas don't come every 3 digits. In india, they come every two digits, after the first three. Thus one billion = 1,00,00,00,000. How are you gonna represent *that* in a formatting mini-language?

Re: [Python-Dev] Formatting mini-language suggestion

2009-03-11 Thread Raymond Hettinger
[Guido van Rossum] I suggest moving this to python-ideas and writing a proper PEP. Okay, it's moved. Will write up a PEP, do research on what other languages do and collect everyone's ideas on what to put in the shed. (hundreds and ten thousands grouping, various choices of decimal

Re: [Python-Dev] Formatting mini-language suggestion

2009-03-11 Thread Guido van Rossum
On Wed, Mar 11, 2009 at 8:34 PM, Raymond Hettinger pyt...@rcn.com wrote:  I expect that nobody likes that idea, Do you mean the idea of a thousands separator or the idea of also parameterizing the decimal point or both? Sorry, neither. I meant the idea of having to write a PEP. :-) (Added

Re: [Python-Dev] Formatting mini-language suggestion

2009-03-11 Thread James Y Knight
On Mar 11, 2009, at 11:40 PM, Nick Coghlan wrote: Raymond Hettinger wrote: It is not the goal to replace locale or to accomodate every possible convention. The goal is to make a common task easier for many users. The current, default use of the period as a decimal point has not proven to be

Re: [Python-Dev] Formatting mini-language suggestion

2009-03-11 Thread Lie Ryan
James Y Knight wrote: On Mar 11, 2009, at 11:40 PM, Nick Coghlan wrote: Raymond Hettinger wrote: It is not the goal to replace locale or to accomodate every possible convention. The goal is to make a common task easier for many users. The current, default use of the period as a decimal point