Re: [Python-Dev] Ext4 data loss
On 11Mar2009 02:20, Antoine Pitrou wrote: | Christian Heimes cheimes.de> writes: | > I agree with you, fsync() shouldn't be called by default. I didn't plan | > on adding fsync() calls all over our code. However I like to suggest a | > file.sync() method and a synced flag for files to make the job of | > application developers easier. | | We already have os.fsync() and os.fdatasync(). Should the sync() (and | datasync()?) method be added as an object-oriented convenience? I can imagine plenty of occasions when there may not be an available file descriptor to hand to os.fsync() et al. Having sync() and datasync() methods in the object would obviate the need for the caller to know the object internals. -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ I must construct my own System, or be enslaved to another Man's. - William Blake ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Ext4 data loss
On 10Mar2009 22:14, A.M. Kuchling wrote: | On Wed, Mar 11, 2009 at 11:31:52AM +1100, Cameron Simpson wrote: | > On 10Mar2009 18:09, A.M. Kuchling wrote: | > | The mailbox module tries to be careful and always fsync() before | > | closing files, because mail messages are pretty important. | > | > Can it be turned off? I hadn't realised this. | | No, there's no way to turn it off (well, you could delete 'fsync' from | the os module). Ah. For myself, were I writing a high load mailbox tool (eg a mail filer or more to the point, a mail refiler - which I do actually intend to) I would want to be able to do a huge mass of mailbox stuff and then possibly issue a sync at the end. For "unix mbox" that might be ok but for maildirs I'd imagine it leads to an fsync per message. | > | The tarfile, zipfile, and gzip/bzip2 classes don't seem to use fsync() | > | at all, either implicitly or by having methods for calling them. | > | Should they? What about cookielib.CookieJar? | > | > I think they should not do this implicitly. By all means let a user | > issue policy. | | The problem is that in some cases the user can't issue policy. For | example, look at dumbdbm._commit(). It renames a file to a backup, | opens a new file object, writes to it, and closes it. A caller can't | fsync() because the file object is created, used, and closed | internally. With zipfile, you could at least access the .fp attribute | to sync it (though is the .fp documented as part of the interface?). I didn't so much mean giving the user an fsync hook so much as publishing a flag such as ".do_critical_fsyncs" inside the dbm or zipfile object. If true, issue fsyncs at appropriate times. | In other words, do we need to ensure that all the relevant library | modules expose an interface to allow requesting a sync, or getting the | file descriptor in order to sync it? With a policy flag you could solve the control issue even for things which don't expose the fd such as your dumbdbm._commit() example. If you supply both a flag and an fsync() method it becomes easy for a user of a module to go: obj = get_dbm_handle() obj.do_critical_fsyncs = False ... do lots and lots of stuff ... obj.fsync() obj.close() Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ In the end, winning is the only safety. - Kerr Avon ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Ext4 data loss
On Tue, Mar 10, 2009 at 7:45 PM, Christian Heimes wrote:
> Antoine Pitrou wrote:
>> Christian Heimes cheimes.de> writes:
>>> I agree with you, fsync() shouldn't be called by default. I didn't plan
>>> on adding fsync() calls all over our code. However I like to suggest a
>>> file.sync() method and a synced flag for files to make the job of
>>> application developers easier.
>>
>> We already have os.fsync() and os.fdatasync(). Should the sync() (and
>> datasync()?) method be added as an object-oriented convenience?
>
> It's more than an object oriented convenience. fsync() takes a file
> descriptor as argument. Therefore I assume fsync() only syncs the data
> to disk that was written to the file descriptor. [*] In Python 2.x we
> are using a FILE* based stream. In Python 3.x we have our own buffered
> writer class.
>
> In order to write all data to disk the FILE* stream must be flushed
> first before fsync() is called:
>
> PyFileObject *f;
> if (fflush(f->f_fp) != 0) {
> /* report error */
> }
> if (fsync(fileno(f->f_fp)) != 0) {
> /* report error */
> }
Let's not think too Unix-specific. If we add such an API it should do
something on Windows too -- the app shouldn't have to test for the
presence of the API. (And thus the API probably shouldn't be called
fsync.)
> Christian
>
> [*] Is my assumption correct, anybody?
It seems to be, at least it's ambiguous.
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Ext4 data loss
Antoine Pitrou wrote:
> Christian Heimes cheimes.de> writes:
>> I agree with you, fsync() shouldn't be called by default. I didn't plan
>> on adding fsync() calls all over our code. However I like to suggest a
>> file.sync() method and a synced flag for files to make the job of
>> application developers easier.
>
> We already have os.fsync() and os.fdatasync(). Should the sync() (and
> datasync()?) method be added as an object-oriented convenience?
It's more than an object oriented convenience. fsync() takes a file
descriptor as argument. Therefore I assume fsync() only syncs the data
to disk that was written to the file descriptor. [*] In Python 2.x we
are using a FILE* based stream. In Python 3.x we have our own buffered
writer class.
In order to write all data to disk the FILE* stream must be flushed
first before fsync() is called:
PyFileObject *f;
if (fflush(f->f_fp) != 0) {
/* report error */
}
if (fsync(fileno(f->f_fp)) != 0) {
/* report error */
}
Christian
[*] Is my assumption correct, anybody?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecated __cmp__ and total ordering
[Michael Foord] > Is there something you don't like about this one: http://code.activestate.com/recipes/576529/ [Mart Sõmermaa ] Yes -- it is not in the standard library. As I said, eventually all the 15,000 matches on Google Code need to update their code and copy that snippet to their util/, write tests for it etc. FWIW, my version is http://code.activestate.com/recipes/576685/ If you want to push for inclusion in the standard library, I would support your effort. The basic idea isn't controversial, but there probably would be a lengthy discussion on what to call it (total_ordering is one possibilty) and where to put it (functools is a possibility). Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Ext4 data loss
Christian Heimes cheimes.de> writes: > > I agree with you, fsync() shouldn't be called by default. I didn't plan > on adding fsync() calls all over our code. However I like to suggest a > file.sync() method and a synced flag for files to make the job of > application developers easier. We already have os.fsync() and os.fdatasync(). Should the sync() (and datasync()?) method be added as an object-oriented convenience? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Ext4 data loss
On Wed, Mar 11, 2009 at 11:31:52AM +1100, Cameron Simpson wrote: > On 10Mar2009 18:09, A.M. Kuchling wrote: > | The mailbox module tries to be careful and always fsync() before > | closing files, because mail messages are pretty important. > > Can it be turned off? I hadn't realised this. No, there's no way to turn it off (well, you could delete 'fsync' from the os module). > | The tarfile, zipfile, and gzip/bzip2 classes don't seem to use fsync() > | at all, either implicitly or by having methods for calling them. > | Should they? What about cookielib.CookieJar? > > I think they should not do this implicitly. By all means let a user > issue policy. The problem is that in some cases the user can't issue policy. For example, look at dumbdbm._commit(). It renames a file to a backup, opens a new file object, writes to it, and closes it. A caller can't fsync() because the file object is created, used, and closed internally. With zipfile, you could at least access the .fp attribute to sync it (though is the .fp documented as part of the interface?). In other words, do we need to ensure that all the relevant library modules expose an interface to allow requesting a sync, or getting the file descriptor in order to sync it? --amk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Ext4 data loss
Guido van Rossum wrote:
> If I understand the post properly, it's up to the app to call fsync(),
> and it's only necessary when you're doing one of the rename dances, or
> updating a file in place. Basically, as he explains, fsync() is a very
> heavyweight operation; I'm against calling it by default anywhere.
I agree with you, fsync() shouldn't be called by default. I didn't plan
on adding fsync() calls all over our code. However I like to suggest a
file.sync() method and a synced flag for files to make the job of
application developers easier.
When a file is opened for writing and has the synced flag set, fsync()
is called immediately before the FILE *fp is closed.
Suggested syntax:
>>> f = open("somefile", "ws")
>>> f.synced
True
or:
>>> f = open(somefile, "w")
>>> f.synced
False
>>> f.synced = True
>>> f.synced
True
The sync() method of a file object flushes the internal buffer(fflush()
for Python 2's file object) and fsync() the file descriptor.
Finally the documentation should give the user a hint that close() does
not necessarily mean the data is written to disk. It's not our
responsibility to teach Python users how to deal with low level stuff.
On the other hand a short warning doesn't hurt us and may help Python
users to write better programs.
For the rest I concur with MvL and AMK.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Ext4 data loss
On 10Mar2009 18:09, A.M. Kuchling wrote: | On Tue, Mar 10, 2009 at 09:11:38PM +0100, Christian Heimes wrote: | > Python's file type doesn't use fsync() and be the victim of the very | > same issue, too. Should we do anything about it? IMHO, beyond _offering_ an fsync method, no. | The mailbox module tries to be careful and always fsync() before | closing files, because mail messages are pretty important. Can it be turned off? I hadn't realised this. | The | various *dbm modules mostly have .sync() method. | | dumbdbm.py doesn't call fsync(), AFAICT; _commit() writes stuff and | closes the file, but doesn't call fsync(). | | sqlite3 doesn't have a sync() or flush() call. Does SQLite handle | this itself? Yeah, most obnoxiously. There's a longstanding firefox bug about the horrendous performance side effects of sqlite's zeal in this regard: https://bugzilla.mozilla.org/show_bug.cgi?id=421482 At least there's now an (almost undocumented) preference to disable it, which I do on a personal basis. | The tarfile, zipfile, and gzip/bzip2 classes don't seem to use fsync() | at all, either implicitly or by having methods for calling them. | Should they? What about cookielib.CookieJar? I think they should not do this implicitly. By all means let a user issue policy. In case you hadn't guessed, I fall into the "never fsync" group, something of a simplification of my real position. In my opinion, deciding to fsync is almost always a user policy decision, not an app decision. An app talks to the OS; if the OS' filesystem has accepted responsibility for the data (as it has after a successful fflush, for example) then normally the app should have no further responsibility; that is _exactly_ what the OS is responsible for. Recovery is what backups are for, generally speaking. All this IMHO, of course. Of course there are some circumstances where one might fsync, as part of one's risk mitigation policies (eg database checkpointing etc). But whenever you do this you're basicly saying you don't trust the OS abstraction of the hardware and also imposing an inherent performance bottleneck. With things like ext3 (and ext4 may well be the same - I have not checked) an fsync doesn't just sync that file data and metadata, it does a whole-filesystem sync. Really expensive. If underlying libraries do that quietly and without user oversight/control then this failure to trust the OS puts an unresolvable bottlneck on various things, and as an app scales up in I/O or operational throughput or as a library or facility becomes "higher level" (i.e. _involving_ more and more underlying complexity or number of basic operations) the more intrusive and unfixable such a low level "auto-fsync" would become. Also, how far do you want to go to assure integrity for particular filesystems' integrity issues/behaviours? Most filesystems sync to disc regularly (or frequently, at any rate) anyway. What's too big a window of potential loss? For myself, I'm against libraries that implicitly do fsyncs, especially if the user can't issue policy about it. Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ If it can't be turned off, it's not a feature. - Karl Heuer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of further status options to tracker
[adding python-dev back on to the email] On Tue, Mar 10, 2009 at 15:51, Tennessee Leeuwenburg wrote: > >>> Pretty much. I've got two views. One is that I'd like to search for >>> issues that are up for grabs which I could take over, hack on, and generally >>> not get underfoot of core development activity. >>> >> >> OK, let's do what is necessary to flag issues like this so that people can >> do this. That's why I like the "Under Development" status. Could rename >> "open" to "available" or "unsolved" to more clearly mark those issues as up >> for grabs. >> > > Yep. I like that too. > > >> Typically we use nosy lists to get specific people's attention. That or >> the priority gets bumped if it turns out to be an important issue. Lastly, >> people can email python-dev directly asking for input if all other attempts >> to get attention have failed. >> > > Now that I am understanding the tracker system better, I think it's fine to > just add reviews to the tracker issue and that will be enough to grab > attention. There is always the option of emailing the list. > > >> >> In other words you want some way to flag an issue that just needs to be >> talked about but is not ready to be coded. So status would go "open/new" -> >> "chatting/needs help" -> "under dev" -> "closed" with "pending" fitting in >> when necessary. Sound about right? >> >> My worry with this is making sure the field gets updated. >> > > Sounds exactly right. I'm not so concerned about this field being updated. > If it doesn't, but someone is clearly working on it, then it's not really > holding anyone back. Tracker janitors (although I do prefer gardeners!) can > worry about whether the field is set correctly, and developers can just get > on with doing their work. > > I like: >"Open/New" >"Needs help / Chatting" >"Under development" >"Pending feedback" >"Closed" > > very much. > > As long as "Under Dev" and "Pending" are time-based to switch to "chatting" or "closed" respectively, I am fine with this. What do other people think? Too heavy-handed? Just right to help people get people involved? -Brett > ...I can understand that, but I would worry that if we flag >> something as under development it will simply be ignored by other people >> when they could actually help out (write the docs for someone, etc.). Or >> even worse that someone gets to a certain point with the development, walks >> away short of finishing the work (say doesn't get the docs finished) and >> everyone continues to ignore the issue because it is still flagged as under >> development. > > > >> If we can come up with a simple solution to this problem (perhaps have >> issues set to under development with no activity shift down a status level >> after a month) then maybe we will have something everyone can be happy with. > > > > Maybe we can just revert anything that is under development back to "needs > help" after a month of inactivity? > > Cheers, > -T > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Ext4 data loss
Neil Hodgson gmail.com> writes: > > What would be useful is a simple, generic > way in Python to copy all the appropriate metadata (ownership, ACLs, > ...) to another file so the temporary-and-rename technique could be > used. How about shutil.copystat()? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Fwd: Addition of further status options to tracker
On Wed, Mar 11, 2009 at 9:43 AM, Brett Cannon wrote: > > > On Tue, Mar 10, 2009 at 15:33, Nick Coghlan wrote: > >> Brett Cannon wrote: >> > If we can come up with a simple solution to this problem (perhaps have >> > issues set to under development with no activity shift down a status >> > level after a month) then maybe we will have something everyone can be >> > happy with. >> >> If an issue is assigned, then somebody has claimed it and is working on >> it (or it has at least been sent to a specific person for consideration). >> > > Right, but that is for core developers only. I think Tennessee is worrying > about non-core folks. > Absolutely, I don't have any issue with the way the most important issues are being worked on now, I just think that less-experienced developers could do a lot more to help out with simple tasks / early-stage tasks. If the rest of us can help ease the burden by getting issues properly sorted out before they go to core developers (writing unit tests, sorting out requirements clearly, documentation, patch suggestions etc) then they won't need to spend as much time on simple maintenance. Cheers, -T -- -- Tennessee Leeuwenburg http://myownhat.blogspot.com/ "Don't believe everything you think" ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Regexp 2.7
Hello, > So, if there's interest in investigating how much of the PCRE tests > can augment the existing tests, I am offering to do so. IMO there's nothing wrong with having more tests, provided that: - they don't make the test suite slower than it should be - they aren't too implementation-specific By the way, having tests is good, but having well-organized and documented tests is even better (something which caught us when working on the io rewrite). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of further status options to tracker
On Tue, Mar 10, 2009 at 15:33, Nick Coghlan wrote: > Brett Cannon wrote: > > If we can come up with a simple solution to this problem (perhaps have > > issues set to under development with no activity shift down a status > > level after a month) then maybe we will have something everyone can be > > happy with. > > If an issue is assigned, then somebody has claimed it and is working on > it (or it has at least been sent to a specific person for consideration). > Right, but that is for core developers only. I think Tennessee is worrying about non-core folks. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of further status options to tracker
Brett Cannon wrote: > If we can come up with a simple solution to this problem (perhaps have > issues set to under development with no activity shift down a status > level after a month) then maybe we will have something everyone can be > happy with. If an issue is assigned, then somebody has claimed it and is working on it (or it has at least been sent to a specific person for consideration). Deciding that someone has dropped off the face of the planet and that their issues should be unassigned is something that a human tracker janitor would probably have to decide to do (the original assignee would get email about the assignment changes and have the option of reclaiming issues they still cared about). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Regexp 2.7
Terry Reedy wrote: > There is a conflict between running a thorough test of everything > possible and not having the test suite run for hours. I believe a > couple of other modules have a regular sanity-check test and an extended > patch-check test. Something like that might be appropriate for re. Correct - a new regrtest resource (e.g. -uregex) could be added if the enhanced regex tests took too much time to run. (-udecimal and -ucompiler are the two existing examples of this that you're likely thinking off) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Ext4 data loss
On Tue, Mar 10, 2009 at 09:11:38PM +0100, Christian Heimes wrote: > Python's file type doesn't use fsync() and be the victim of the very > same issue, too. Should we do anything about it? The mailbox module tries to be careful and always fsync() before closing files, because mail messages are pretty important. The various *dbm modules mostly have .sync() method. dumbdbm.py doesn't call fsync(), AFAICT; _commit() writes stuff and closes the file, but doesn't call fsync(). sqlite3 doesn't have a sync() or flush() call. Does SQLite handle this itself? The tarfile, zipfile, and gzip/bzip2 classes don't seem to use fsync() at all, either implicitly or by having methods for calling them. Should they? What about cookielib.CookieJar? --amk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Ext4 data loss
> If I understand the post properly, it's up to the app to call fsync(), Correct. > and it's only necessary when you're doing one of the rename dances, or > updating a file in place. No. It's in general necessary when you want to be sure that the data is on disk, even if the power is lost. So even if you write a file (say, a .pyc) only once - if the lights go out, and on again, your .pyc might be corrupted, as the file system may have chosen to flush the metadata onto disk, but not the actual data (or only parts of it). This may happen even if the close(2) operation was successful. In the specific case of config files, that's unfortunate because you then can't revert to the old state, either - because that may be gone. Ideally, you want transactional updates - you get either the old config or the new config after a crash. You can get that with explicit fdatasync, or with a transactional database (which may chose to sync only infrequently, but then will be able to rollback the old state if the new one wasn't written completely). But yes, I agree, it's the applications' responsibility to properly sync. If I had to place sync calls into the standard library, they would go into dumbdbm. I somewhat disagree that it is the application's fault entirely, and not the operating system's/file system's fault. Ideally, there would be an option of specifying transaction brackets for file operations, so that the system knows it cannot flush the unlink operation of the old file before it has flushed the data of the new file. This would still allow the system to schedule IO fairly freely, but also guarantee that not all gets lost in a crash. I thought that the data=ordered ext3 mount option was going in that direction - not sure what happened to it in ext4. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] More on Py3K urllib -- urlencode()
Aahz wrote: > Second, please follow the advice to put ALL patches on the tracker. I don't care about top-posting, but I second the Second point. Let's move this thread to the issue tracker. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Ext4 data loss
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 10, 2009, at 4:46 PM, Neil Hodgson wrote: The technique advocated by Theodore Ts'o (save to temporary then rename) discards metadata. What would be useful is a simple, generic way in Python to copy all the appropriate metadata (ownership, ACLs, ...) to another file so the temporary-and-rename technique could be used. On Windows, there is a hack in the file system that tries to track the use of temporary-and-rename and reapply ACLs and on OS X there is a function FSPathReplaceObject but I don't know how to do this correctly on Linux. Of course, a careful *nix application can ensure that the file owners and mod bits are set the way it needs them to be set. A convenience function might be useful though. Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSbbb8XEjvBPtnXfVAQLxvgP/SDnwzcKHI9E9K/ShAVWk3aShsDvJSztH wHRQlOkbxxG/xcGJ7hGYaxJh5/TszU4wvtSc7JV5p6tRWrk/FRvAPW9lFBrlVQ8I ZTV/bsNRJLSDxEXe7H4S2/c0L8LuGu58RGWtQzFH0UlnIFYIDwxxVGjfpVckXAc4 l54OAhDPFSk= =njh4 -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Ext4 data loss
On Tue, Mar 10, 2009 at 1:46 PM, Neil Hodgson wrote: > The technique advocated by Theodore Ts'o (save to temporary then > rename) discards metadata. What would be useful is a simple, generic > way in Python to copy all the appropriate metadata (ownership, ACLs, > ...) to another file so the temporary-and-rename technique could be > used. > > On Windows, there is a hack in the file system that tries to track > the use of temporary-and-rename and reapply ACLs and on OS X there is > a function FSPathReplaceObject but I don't know how to do this > correctly on Linux. I don't know how to implement this for metadata beyond the traditional stat metadata, but the API could be an extension of shutil.copystat(). -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Ext4 data loss
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 10, 2009, at 4:23 PM, Guido van Rossum wrote: On Tue, Mar 10, 2009 at 1:11 PM, Christian Heimes wrote: Multiple blogs and news sites are swamped with a discussion about ext4 and KDE 4.0. Theodore Ts'o - the developer of ext4 - explains the issue at https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/54 . Python's file type doesn't use fsync() and be the victim of the very same issue, too. Should we do anything about it? If I understand the post properly, it's up to the app to call fsync(), and it's only necessary when you're doing one of the rename dances, or updating a file in place. Basically, as he explains, fsync() is a very heavyweight operation; I'm against calling it by default anywhere. Right. Python /applications/ should call fsync() and do the rename dance if appropriate, and fortunately it's easy enough to implement in Python. Mailman's queue runner has done exactly this for ages. Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iQCUAwUBSbbSXHEjvBPtnXfVAQLrsQP2NxJL+js6fMDgluoSpB6kW+VCJfSS0G58 KaDiRniinl3E9qH9w+hvNE7Es9JzPSiOP79KkuqRkzvCCmkrQRvsY6dKukOs/1zq KNpTB4I3bGzUHgM+OwAh2KuxJ3pXzOPhrPwLLXLq7k1OuGRODmPxWXZ+i8FirR7C 8fpV6wNFAQ== =JIdS -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Ext4 data loss
The technique advocated by Theodore Ts'o (save to temporary then rename) discards metadata. What would be useful is a simple, generic way in Python to copy all the appropriate metadata (ownership, ACLs, ...) to another file so the temporary-and-rename technique could be used. On Windows, there is a hack in the file system that tries to track the use of temporary-and-rename and reapply ACLs and on OS X there is a function FSPathReplaceObject but I don't know how to do this correctly on Linux. Neil ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Regexp 2.7
Jared Grubb wrote: I'm not criticizing the current battery of tests, nor am I arguing that we replace them. There's a comment in the test_re.py that says that "these tests were carefully modeled to cover most of the code"... That is a very difficult statement to maintain and/or verify, especially if the library gets a major revision (which it appears the original post's patch is). PCRE has _thousands_ of detailed regular expression tests, testing everything from matching to parsing to extended regular expression syntax to encoding and locales. (It's been a while since I've looked at the details, but of course there are tests that dont apply to Python's implmentation.) So, if there's interest in investigating how much of the PCRE tests can augment the existing tests, I am offering to do so. (I already did a simple translation utility to parse the PCRE test format into something we could use in the PyPy test suite; I could try to do something similar for test_re, if there's interest). There is a conflict between running a thorough test of everything possible and not having the test suite run for hours. I believe a couple of other modules have a regular sanity-check test and an extended patch-check test. Something like that might be appropriate for re. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Ext4 data loss
On Tue, Mar 10, 2009 at 1:11 PM, Christian Heimes wrote: > Multiple blogs and news sites are swamped with a discussion about ext4 > and KDE 4.0. Theodore Ts'o - the developer of ext4 - explains the issue > at > https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/54. > > > Python's file type doesn't use fsync() and be the victim of the very > same issue, too. Should we do anything about it? If I understand the post properly, it's up to the app to call fsync(), and it's only necessary when you're doing one of the rename dances, or updating a file in place. Basically, as he explains, fsync() is a very heavyweight operation; I'm against calling it by default anywhere. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Ext4 data loss
Multiple blogs and news sites are swamped with a discussion about ext4 and KDE 4.0. Theodore Ts'o - the developer of ext4 - explains the issue at https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/54. Python's file type doesn't use fsync() and be the victim of the very same issue, too. Should we do anything about it? Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] reviewing patches
On Tue, Mar 10, 2009 at 11:43, Terry Reedy wrote: > Brett Cannon wrote: > >> >> This is somewhat covered by components, but it's implicit. Would it be >> worth making this explicit? I have always wondered if people would be more >> willing to help out if they could easily search for pure Python code issues >> if that is as far as they feel comfortable. >> > > If and when I am ready to move from working on documentation issues (which > seem to becoming fewer as the 3.x transition is completed) to code issues, > that would be helpful. What would be really helpful is to have library > issues tagged and sorted by specific modules (or modules, if more than one), > but I do not know how that might be done. It would have to be a text field that people just filled in. Making a list that had to be kept up-to-date would be a disaster. But one would hope that simply searching for an issue with a specific module name would be enough to warrant not having to have the field. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Regexp 2.7
I'm not criticizing the current battery of tests, nor am I arguing that we replace them. There's a comment in the test_re.py that says that "these tests were carefully modeled to cover most of the code"... That is a very difficult statement to maintain and/or verify, especially if the library gets a major revision (which it appears the original post's patch is). PCRE has _thousands_ of detailed regular expression tests, testing everything from matching to parsing to extended regular expression syntax to encoding and locales. (It's been a while since I've looked at the details, but of course there are tests that dont apply to Python's implmentation.) So, if there's interest in investigating how much of the PCRE tests can augment the existing tests, I am offering to do so. (I already did a simple translation utility to parse the PCRE test format into something we could use in the PyPy test suite; I could try to do something similar for test_re, if there's interest). Jared On 10 Mar 2009, at 11:32, Guido van Rossum wrote: Hm, what's wrong with the existing set of regex test cases? This is one of the most complete set of test cases in our test suite. On Tue, Mar 10, 2009 at 11:08 AM, Jared Grubb wrote: Would there be any interest in augmenting the test case library for the regex stuff? On 9 Mar 2009, at 16:07, Antoine Pitrou wrote: Facundo Batista gmail.com> writes: Matthew Barnett has been doing a lot of work on the regular expressions engine (it seems he hasn't finished yet) under http://bugs.python.org/issue2636 . However, the patches are really huge and touch all of the sre internals. I wonder what the review process can be for such patches? Is there someone knowledgeable enough to be able to review them? All test cases run ok? How well covered is that library? I don't know, I haven't even tried it. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Regexp 2.7
On Tue, Mar 10, 2009 at 11:32:10AM -0700, Guido van Rossum wrote: > Hm, what's wrong with the existing set of regex test cases? This is > one of the most complete set of test cases in our test suite. There's never anything wrong with having more test cases! However, if you have a choice of which test suite to improve, I would choose another module to work on. The re module was once based on PCRE and a lot of its test suite was ported into the Python one, so there may not much relevant to add. But please do whatever you like... --amk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] More on Py3K urllib -- urlencode()
On Tue, Mar 10, 2009, Dan Mahn wrote: > > Ahh ... I see. I should have done a bit more digging to find where the > standard tests were. > > I created a few new tests that could be included in that test suite -- > see the attached file. Do you think that this would be sufficient? First of all, please notice from the list traffic that except for Guido (who gets special dispensation because he's BDFL), most messages do not use top-posting: A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet? Second, please follow the advice to put ALL patches on the tracker. -- Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/ "All problems in computer science can be solved by another level of indirection." --Butler Lampson ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] More on Py3K urllib -- urlencode()
Ahh ... I see. I should have done a bit more digging to find where the
standard tests were.
I created a few new tests that could be included in that test suite --
see the attached file. Do you think that this would be sufficient?
- Dan
Bill Janssen wrote:
Dan Mahn wrote:
Yes, that was a good idea. I found some problems, and attached a new
version. It looks more complicated than I wanted, but it is a very
regular repetition, so I hope it is generally readable.
That's great, but I was hoping for more tests in lib/test/test_urllib.py.
Bill
def test_encoding(self):
# Test for special character encoding
given = (("\u00a0", "\u00c1"),)
expect = '%3F=%3F'
result = urllib.parse.urlencode(given, encoding="ASCII",
errors="replace")
self.assertEqual(expect, result)
result = urllib.parse.urlencode(given, True, encoding="ASCII",
errors="replace")
self.assertEqual(expect, result)
given = (("\u00a0", (1, "\u00c1")),)
# ... now with default utf-8 ...
given = (("\u00a0", "\u00c1"),)
expect = '%C2%A0=%C3%81'
result = urllib.parse.urlencode(given)
self.assertEqual(expect, result)
# ... now with latin-1 ...
expect = '%A0=%C1'
result = urllib.parse.urlencode(given, encoding="latin-1")
self.assertEqual(expect, result)
# ... now with sequence ...
given = (("\u00a0", (1, "\u00c1")),)
expect = '%3F=1&%3F=%3F'
result = urllib.parse.urlencode(given, True, encoding="ASCII",
errors="replace")
self.assertEqual(expect, result)
# ... again with default utf-8 ...
given = (("\u00a0", "\u00c1"),)
expect = '%C2%A0=%C3%81'
result = urllib.parse.urlencode(given, True)
self.assertEqual(expect, result)
# ... again with latin-1 ...
expect = '%A0=%C1'
result = urllib.parse.urlencode(given, True, encoding="latin-1")
self.assertEqual(expect, result)
def test_bytes(self):
# Test for encoding bytes
given = ((b'\xa0\x24', b'\xc1\x24'),)
expect = '%A0%24=%C1%24'
result = urllib.parse.urlencode(given)
self.assertEqual(expect, result)
# ... now with sequence ...
result = urllib.parse.urlencode(given, True)
self.assertEqual(expect, result)
# ... now with safe and encoding ...
expect = '%A0$=%C1$'
result = urllib.parse.urlencode(given, safe=":$")
self.assertEqual(expect, result)
result = urllib.parse.urlencode(given, safe=":$", encoding="latin-1")
self.assertEqual(expect, result)
# ... again with sequence ...
result = urllib.parse.urlencode(given, True, safe=":$")
self.assertEqual(expect, result)
result = urllib.parse.urlencode(given, True, safe=":$",
encoding="latin-1")
self.assertEqual(expect, result)
# ... now with an actual sequence ...
given = ((b'\xa0\x24', (b'\xc1\x24', 0xd)),)
result = urllib.parse.urlencode(given, True, safe=":$")
self.assert_(expect in result,
"%s not found in %s" % (expect, result))
expect2 = '%A0$=1'
self.assert_(expect2 in result,
"%s not found in %s" % (expect2, result))
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] reviewing patches
Brett Cannon wrote: This is somewhat covered by components, but it's implicit. Would it be worth making this explicit? I have always wondered if people would be more willing to help out if they could easily search for pure Python code issues if that is as far as they feel comfortable. If and when I am ready to move from working on documentation issues (which seem to becoming fewer as the 3.x transition is completed) to code issues, that would be helpful. What would be really helpful is to have library issues tagged and sorted by specific modules (or modules, if more than one), but I do not know how that might be done. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Regexp 2.7
Hm, what's wrong with the existing set of regex test cases? This is one of the most complete set of test cases in our test suite. On Tue, Mar 10, 2009 at 11:08 AM, Jared Grubb wrote: > Would there be any interest in augmenting the test case library for the > regex stuff? > > When I was working on PyPy, we were using a simplified regular expression > matcher to implement the tokenizer for Python. I was able to take a lot of > PCRE's regex tests and port them to test our regular expression > implementation (to make sure the DFA's were being optimized properly, etc). > > I believe the PCRE test library was under a very liberal license, and so we > may be able to do the same here. If there's interest in it, I can do the > same for Python. > > Jared > > On 9 Mar 2009, at 16:07, Antoine Pitrou wrote: > >> Facundo Batista gmail.com> writes: >>> Matthew Barnett has been doing a lot of work on the regular expressions >> >> engine (it seems he hasn't finished yet) under http://bugs.python.org/issue2636. However, the patches are really huge and touch all of the sre internals. I wonder what the review process can be for such patches? Is there someone knowledgeable enough to be able to review them? >>> >>> All test cases run ok? How well covered is that library? >> >> I don't know, I haven't even tried it. >> >> Regards >> Antoine. > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Regexp 2.7
Would there be any interest in augmenting the test case library for the regex stuff? When I was working on PyPy, we were using a simplified regular expression matcher to implement the tokenizer for Python. I was able to take a lot of PCRE's regex tests and port them to test our regular expression implementation (to make sure the DFA's were being optimized properly, etc). I believe the PCRE test library was under a very liberal license, and so we may be able to do the same here. If there's interest in it, I can do the same for Python. Jared On 9 Mar 2009, at 16:07, Antoine Pitrou wrote: Facundo Batista gmail.com> writes: Matthew Barnett has been doing a lot of work on the regular expressions engine (it seems he hasn't finished yet) under http://bugs.python.org/issue2636 . However, the patches are really huge and touch all of the sre internals. I wonder what the review process can be for such patches? Is there someone knowledgeable enough to be able to review them? All test cases run ok? How well covered is that library? I don't know, I haven't even tried it. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyS60 - mailbox
On Tue, 10 Mar 2009 at 22:41, Mahesh S wrote: I am from the PyS60 (Python port on S60 platform) team. We have ported the Python 2.5.4 core in the latest 1.9.x series of PyS60. http://wiki.opensource.nokia.com/projects/PyS60 Currently we have a problem with the mailbox module. Check the code snippet below, from the mailbox module. *def clean(self): """Delete old files in "tmp".""" now = time.time() for entry in os.listdir(os.path.join(self._**path, 'tmp')): path = os.path.join(self._path, 'tmp', entry) if now - os.path.getatime(path) > 129600: # 60 * 60 * 36 os.remove(path)* The code in red, tries to delete the files that are not accessed in the last 36 hours. Any idea as to how this is supposed to work on platforms that do not support access time, for example Symbian/S60 ? Any work around ? Many unix systems (especially laptops) now run with the 'noatime' option set on their file systems. On my system with noatime set, atime appears to be equal to mtime, so assuming you have mtime, returning that for atime would appear to be de facto standards compliant. -- R. David Murray http://www.bitdance.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecated __cmp__ and total ordering
Mart Sõmermaa wrote: On Tue, Mar 10, 2009 at 3:57 PM, Michael Foord mailto:fuzzy...@voidspace.org.uk>> wrote: Is there something you don't like about this one: http://code.activestate.com/recipes/576529/ Yes -- it is not in the standard library. As I said, eventually all the 15,000 matches on Google Code need to update their code and copy that snippet to their util/, write tests for it etc. That question was in reply to Raymond who said he was working on his own version. Michael ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OS X Installer for 3.0.1 and supported versions
In article ,
Ronald Oussoren wrote:
> On 27 Feb, 2009, at 1:57, Ned Deily wrote:
>
> > In article ,
> > "Russell E. Owen" wrote:
> >> I want to follow up on this a bit. In the past if the Mac Python
> >> installer was built on a machine that did NOT have a locally
> >> installed
> >> Tcl/Tk then it would fail to work with a locally installed Tcl/Tk:
> >> Python would segfault when trying to use Tkinter.
> >>
> >> The solution was to build the Mac python installer on a machine
> >> with a
> >> locally installed Tcl/Tk. The resulting installer package would
> >> work on
> >> all systems -- with or without locally installed Tcl/Tk.
> >>
> >> So...has this problem been worked around, or is the Mac installer
> >> still
> >> built on a machine that has a locally installed Tcl/Tk?
> >
> > Ronald will have to answer that for sure since he built the installer
> > for 3.0.1.
> >
> > However, it seems to be true that the most recent python.org OS X
> > installers will always favor a /System/Library/Frameworks/{Tcl/Tk}:
>
> That's correct, I don't have a locally installed Tcl/Tk on my laptop
> at the moment and couldn't arrange for one in time for the 3.0.1
> release.
>
> BTW. The fact that this should result in crashes when a user does have
> a locally installed Tcl/Tk is new to me. The reason earlier builds of
> the OSX installer were build with a locally installed Tcl/Tk is that
> several Tkinter users indicated that the system version is
> significantly less useful than a local install.
Any hope of an updated Mac installer for 2.5.4 and/or 2.6 that is built
with a locally installed Tcl/Tk? If/when you do that, I strongly
recommend ActiveState Tcl/Tk 8.4.19 -- the last 8.4 release -- it has
few bugs and very good performance.
(For Python 2.6 you could install Tcl/Tk 8.5 instead, but it might not
work for folks using Apple's built-in Tcl/Tk 8.4. Maybe somebody else
knows; if not it would require some testing.)
Note that even Python 2.5 built against Tcl/Tk 8.5 and most of it
worked, but there were some known bugs.
-- Russell
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyS60 - mailbox
Great news on the port! On your issue: Is this module relevant for the platform? Do you know of any existing app that uses the qmail mailbox format? As a work-around, I can think of several values to substitute for atime: the current time, or the mtime, or perhaps even the max of mtime and ctime. --Guido On Tue, Mar 10, 2009 at 10:11 AM, Mahesh S wrote: > Hi, > > I am from the PyS60 (Python port on S60 platform) team. We have ported the > Python 2.5.4 core in the latest 1.9.x series of PyS60. > http://wiki.opensource.nokia.com/projects/PyS60 > > Currently we have a problem with the mailbox module. Check the code snippet > below, from the mailbox module. > > def clean(self): > """Delete old files in "tmp".""" > now = time.time() > for entry in os.listdir(os.path.join(self._path, 'tmp')): > path = os.path.join(self._path, 'tmp', entry) > if now - os.path.getatime(path) > 129600: # 60 * 60 * 36 > os.remove(path) > > The code in red, tries to delete the files that are not accessed in the last > 36 hours. Any idea as to how this is supposed to work on platforms that do > not support access time, for example Symbian/S60 ? Any work around ? > > - Mahesh > > > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of further status options to tracker
On Mon, Mar 9, 2009 at 22:39, Tennessee Leeuwenburg wrote: > I don't mind what approach is taken -- I'm happy to work within the >>> current infrastructure if someone can suggest a good way. I really just want >>> to be able to start distinguishing between issues that are essentially new >>> and under debate versus issues that most people agree are a "Good Thing To >>> Do", and then helping those issues advance to the point of implementation by >>> discussing and, if necessary, formalising or recording requirements against >>> them. >>> >> >> I have always viewed it that if Stage is set to anything other than its >> empty default it is worth looking into. But it seems to me what you are >> after is some obvious way to disambiguate issues that are feature requests >> that someone has just asked for and anyone can work on if they want, and >> issues that require attention, i.e. bugs and patches. Is that accurate? > > > Pretty much. I've got two views. One is that I'd like to search for issues > that are up for grabs which I could take over, hack on, and generally not > get underfoot of core development activity. > OK, let's do what is necessary to flag issues like this so that people can do this. That's why I like the "Under Development" status. Could rename "open" to "available" or "unsolved" to more clearly mark those issues as up for grabs. > The other is that I think I can help with issue gardening -- splitting > issues up into those which still need some more thought, those which someone > should pick up and start working on, etc. > >* Ideas needing more consideration >* Ideas being hotly debated >* Good ideas with no owning developer >* Ideas currently under initial development >* Ideas ready for consideration by the core developers > > In order to make the most use of experienced developers, I'd say it's > important that they spend most of their time concentrated at the bottom of > that list. Bugs and patches more or less automatically qualify for that, and > they can be searched on pretty effectively now. However, doing gardening on > the issues which aren't quite there yet is currently pretty clumsy. > Typically we use nosy lists to get specific people's attention. That or the priority gets bumped if it turns out to be an important issue. Lastly, people can email python-dev directly asking for input if all other attempts to get attention have failed. > > Say I'm a core developer and very busy. I'd like to quickly address bugs if > possible, and I'm happy to help out on important new issues. I certainly > don't want to trawl through a whole lot of immature issues and do the triage > myself -- what a waste of time! Right now, what can such a person search on > to find the relevant issues? It looks like "patch review" or "commit review" > will do nicely. This isn't going to change, so that is still supported. Bugs > are essentially everything not a feature request, so that doesn't change > either. > > Okay, so the developer workflow is simple and supported. But what about > pre-core development workflow? > > How about for those involved in performing triage and specification? Where > I work, at least 50% of the time is spent just working out what to do. > There's a lot of real work in triage and specification, and it really > shouldn't be done by core developers who could be churning out shiny new > stuff. That means that the triage team (or issue janitors) need to be able > to support a workflow this is pretty easy on them, too. At this stage, we're > not dealing with code, we're just dealing with problems. > In other words you want some way to flag an issue that just needs to be talked about but is not ready to be coded. So status would go "open/new" -> "chatting/needs help" -> "under dev" -> "closed" with "pending" fitting in when necessary. Sound about right? My worry with this is making sure the field gets updated. > > If we want people with great ideas to get to the top of the heap quickly, > we need some way to facilitate that, while issues that are either > inappropriate or being hotly contested don't suck time away from more > important things. > Doesn't seem to have been much of a problem so far. > > Anyway, I really do just want to fit in. I'm just butting into some > workflow things which seem a bit clumsy when doing issue gardening or when > finding coding tasks that are up for grabs for a python-dev newbie... > What I have been telling people is to search for open issues that match up with what you are looking for based on Stage (whether it be docs, testing, writing patches), what it affects (based on Component so you can skip C-related issues) and if it is flagged as easy or not. Your point is that's fine, but that also turns up issues that people are already working on and so people constantly bump up against issues that are being dealt with. I can understand that, but I would worry that if we flag something as under development it will simply be ig
[Python-Dev] PyS60 - mailbox
Hi, I am from the PyS60 (Python port on S60 platform) team. We have ported the Python 2.5.4 core in the latest 1.9.x series of PyS60. http://wiki.opensource.nokia.com/projects/PyS60 Currently we have a problem with the mailbox module. Check the code snippet below, from the mailbox module. *def clean(self): """Delete old files in "tmp".""" now = time.time() for entry in os.listdir(os.path.join(self._**path, 'tmp')): path = os.path.join(self._path, 'tmp', entry) if now - os.path.getatime(path) > 129600: # 60 * 60 * 36 os.remove(path)* The code in red, tries to delete the files that are not accessed in the last 36 hours. Any idea as to how this is supposed to work on platforms that do not support access time, for example Symbian/S60 ? Any work around ? - Mahesh ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecated __cmp__ and total ordering
On Tue, Mar 10, 2009 at 3:57 PM, Michael Foord wrote: > Is there something you don't like about this one: > http://code.activestate.com/recipes/576529/ > Yes -- it is not in the standard library. As I said, eventually all the 15,000 matches on Google Code need to update their code and copy that snippet to their util/, write tests for it etc. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Review of Issue http://bugs.python.org/issue2706, timedelta operators
On Tue, Mar 10, 2009 at 12:34 AM, Tennessee Leeuwenburg wrote: > .. I'm not sure what's best: > * Post the review as a comment on the associated issue. Only nosies will > be updated. You should always do that. If you feel that the nosy list is too small and you want to raise awareness for the issue, you can either CC the list or post a note with a link to the issue. Please keep in mind that tracker summaries are posted on this list every week and updated issues move to the top of the default tracker view, so just posting a new comment will already give it additional visibility. > * Post the review as a comment on the issue, then post a one-line > information update to this list All comments should be posted to the tracker. Comments summarizing long tracker discussions and proposing a resolution may be appropriate for a python-dev post, but if you do that, please reference the list thread on the tracker as it develops here. > * Post the review to this list for consideration Please do this only after a tracker update. This way your review will not be lost. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] More on Py3K urllib -- urlencode()
Dan Mahn wrote: > Yes, that was a good idea. I found some problems, and attached a new > version. It looks more complicated than I wanted, but it is a very > regular repetition, so I hope it is generally readable. That's great, but I was hoping for more tests in lib/test/test_urllib.py. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] More on Py3K urllib -- urlencode()
I submitted an explanation of this and my proposed modification as issue 5468. http://bugs.python.org/issue5468 - Dan Bill Janssen wrote: Aahz wrote: On Sat, Mar 07, 2009, Dan Mahn wrote: After a harder look, I concluded there was a bit more work to be done, but still very basic modifications. Attached is a version of urlencode() which seems to make the most sense to me. I wonder how I could officially propose at least some of these modifications. Submit a patch to bugs.python.org And it would help if it included a lot of test cases. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecated __cmp__ and total ordering
Raymond Hettinger wrote: [Mart Sõmermaa] To provide total ordering without __cmp__ one has to implement all of __lt__, __gt__, __le__, __ge__, __eq__ and __ne__. However, in all but a few cases it suffices only to provide a "real" implementation for e.g. __lt__ and define all the other methods in terms of it as follows: [Raymond Hettinger] FWIW, I'm working on a solution for the problem using class decorators. The idea is that it would scan a class and fill-in missing methods based on the ones already there. That way, any one of the four ordering relations can be provided as a starting point and we won't have to annoint one like __lt__ as the one true underlying method. When it's ready, I'll bring it to python-dev for discussion. [Michael Foord] Is there something you don't like about this one: http://code.activestate.com/recipes/576529/ Yes, that recipe has the basic idea! It was originally written after you issued a challenge at PyCon UK last year. I think the implementation can be cleaned-up quite a bit and it can be made as fast as hand-written code (not the setup time, but the actual introduced method). OK I'll take it back to Christian and Menno and see what we can come up with. Michael Raymond -- http://www.ironpythoninaction.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecated __cmp__ and total ordering
[Mart Sõmermaa] To provide total ordering without __cmp__ one has to implement all of __lt__, __gt__, __le__, __ge__, __eq__ and __ne__. However, in all but a few cases it suffices only to provide a "real" implementation for e.g. __lt__ and define all the other methods in terms of it as follows: [Raymond Hettinger] FWIW, I'm working on a solution for the problem using class decorators. The idea is that it would scan a class and fill-in missing methods based on the ones already there. That way, any one of the four ordering relations can be provided as a starting point and we won't have to annoint one like __lt__ as the one true underlying method. When it's ready, I'll bring it to python-dev for discussion. [Michael Foord] Is there something you don't like about this one: http://code.activestate.com/recipes/576529/ Yes, that recipe has the basic idea! I think the implementation can be cleaned-up quite a bit and it can be made as fast as hand-written code (not the setup time, but the actual introduced method). Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Decimal in C status (was Re: draft 3.1 releaseschedule)
Raymond Hettinger wrote: > > [Nick Coghlan] >>> What about decimal-in-C, by the way? Is anyone still working on it? > > I'm seeking funding for the project. If it is forthcoming, I intend to do > a pure C version that simply implements the spec and then adds > wrappers for the pure python interface. That will save the cost > of constantly creating and modifying many pyobjects that currently > arise during intermediate calculations. > It's possible to focus just of the mantissa calculations but the cost of > the actual arithmetic work is swapped by the overhead of managing > contexts, checking for special values, and memory allocations. > Without addressing those, I think decimal will remain critically > performance challenged compared to native floats (decimals > will never be that fast, but they can get close enough to make > them a viable alternative for many kinds of work). I actually agree with all that, but do you agree the mantissa work is still worthwhile in the near term (i.e. 3.1) to address the 25% or so slowdown between decimal in 2.x (with the mantissa as an 8-bit str) and decimal in 3.0 (with the mantissa as a unicode str, because using a bytes object to store the digits is actually slower due to the lack of a bytes->long fast path)? Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecated __cmp__ and total ordering
Raymond Hettinger wrote: [Mart Sõmermaa] To provide total ordering without __cmp__ one has to implement all of __lt__, __gt__, __le__, __ge__, __eq__ and __ne__. However, in all but a few cases it suffices only to provide a "real" implementation for e.g. __lt__ and define all the other methods in terms of it as follows: FWIW, I'm working on a solution for the problem using class decorators. The idea is that it would scan a class and fill-in missing methods based on the ones already there. That way, any one of the four ordering relations can be provided as a starting point and we won't have to annoint one like __lt__ as the one true underlying method. When it's ready, I'll bring it to python-dev for discussion. Is there something you don't like about this one: http://code.activestate.com/recipes/576529/ Michael Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecated __cmp__ and total ordering
[Mart Sõmermaa] To provide total ordering without __cmp__ one has to implement all of __lt__, __gt__, __le__, __ge__, __eq__ and __ne__. However, in all but a few cases it suffices only to provide a "real" implementation for e.g. __lt__ and define all the other methods in terms of it as follows: FWIW, I'm working on a solution for the problem using class decorators. The idea is that it would scan a class and fill-in missing methods based on the ones already there. That way, any one of the four ordering relations can be provided as a starting point and we won't have to annoint one like __lt__ as the one true underlying method. When it's ready, I'll bring it to python-dev for discussion. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Decimal in C status (was Re: draft 3.1 releaseschedule)
[Nick Coghlan] What about decimal-in-C, by the way? Is anyone still working on it? I'm seeking funding for the project. If it is forthcoming, I intend to do a pure C version that simply implements the spec and then adds wrappers for the pure python interface. That will save the cost of constantly creating and modifying many pyobjects that currently arise during intermediate calculations. It's possible to focus just of the mantissa calculations but the cost of the actual arithmetic work is swapped by the overhead of managing contexts, checking for special values, and memory allocations. Without addressing those, I think decimal will remain critically performance challenged compared to native floats (decimals will never be that fast, but they can get close enough to make them a viable alternative for many kinds of work). Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Decimal in C status (was Re: draft 3.1 release schedule)
Antoine Pitrou wrote: > Raymond Hettinger rcn.com> writes: > > You might also want to collect a list of serious changes that you > want in this release; >> I'm making minor updates to the decimal module to match the 1.68 version of > the spec. > > What about decimal-in-C, by the way? Is anyone still working on it? There was a bit of a difference in philosophy between Mark and Raymond as to how best to proceed with speeding it up (Mark's patch [1] focuses solely on speeding up mantissa manipulation, while Raymond suggested a more extensive rewrite in C would be preferable), so progress on the issue stalled. It would probably require at least a comment from Raymond saying to proceed with the custom mantissa type approach to get that patch moving again. Alternatively, if Raymond actually gets the project sponsorship he mentioned a couple of months ago, that would also get things moving again (just in a different direction). Cheers, Nick. [1] http://bugs.python.org/issue2486 -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecated __cmp__ and total ordering
Hello Mart, This has been discussed before. Guido was against automatically filling in these methods based I think on the fact that this may not be what you want - worth searching the archives for. See here for a class decorator that provides all rich comparison methods for classes that only implement one (e.g. __lt__): http://code.activestate.com/recipes/576529/ Michael Foord Mart Sõmermaa wrote: __cmp__ used to provide a convenient way to make all ordering operators work by defining a single method. For better or worse, it's gone in 3.0. To provide total ordering without __cmp__ one has to implement all of __lt__, __gt__, __le__, __ge__, __eq__ and __ne__. However, in all but a few cases it suffices only to provide a "real" implementation for e.g. __lt__ and define all the other methods in terms of it as follows: class TotalOrderMixin(object): def __lt__(self, other): raise NotImplemented # override this def __gt__(self, other): return other < self def __le__(self, other): return not (other < self) def __ge__(self, other): return not (self < other) __eq__ and __ne__ are somewhat special, although it is possible to define them in terms of __lt__ def __eq__(self, other): return not (self == other) def __ne__(self, other): return self < other or other < self it may be inefficient. So, to avoid the situation where all the projects that match http://www.google.com/codesearch?q=__cmp__+lang%3Apython have to implement their own TotalOrderMixin, perhaps one could be provided in the stdlib? Or even better, shouldn't a class grow automagic __gt__, __le__, __ge__ if __lt__ is provided, and, in a similar vein, __ne__ if __eq__ is provided? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Deprecated __cmp__ and total ordering
__cmp__ used to provide a convenient way to make all ordering operators work by defining a single method. For better or worse, it's gone in 3.0. To provide total ordering without __cmp__ one has to implement all of __lt__, __gt__, __le__, __ge__, __eq__ and __ne__. However, in all but a few cases it suffices only to provide a "real" implementation for e.g. __lt__ and define all the other methods in terms of it as follows: class TotalOrderMixin(object): def __lt__(self, other): raise NotImplemented # override this def __gt__(self, other): return other < self def __le__(self, other): return not (other < self) def __ge__(self, other): return not (self < other) __eq__ and __ne__ are somewhat special, although it is possible to define them in terms of __lt__ def __eq__(self, other): return not (self == other) def __ne__(self, other): return self < other or other < self it may be inefficient. So, to avoid the situation where all the projects that match http://www.google.com/codesearch?q=__cmp__+lang%3Apython have to implement their own TotalOrderMixin, perhaps one could be provided in the stdlib? Or even better, shouldn't a class grow automagic __gt__, __le__, __ge__ if __lt__ is provided, and, in a similar vein, __ne__ if __eq__ is provided? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of further status options to tracker
Tennessee Leeuwenburg gmail.com> writes: > > Hi all,I am beginning reviewing some more issues in the tracker. I think it would be useful to have the following status options (new status options marked with a '+'): I have to point out that the more alternatives there are to choose from, the more choosing from them can feel like a bureaucratic burden. Since Python has no formal development process (as opposed to, say, Twisted), I think the tracker should match today's quite informal and flexible way of working. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] draft 3.1 release schedule
Raymond Hettinger rcn.com> writes: > > >>> You might also want to collect a list of serious changes that you > >>> want in this release; > > I'm making minor updates to the decimal module to match the 1.68 version of the spec. What about decimal-in-C, by the way? Is anyone still working on it? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate lxml into the stdlib?
On Fri, Mar 6, 2009 at 22:10, "Martin v. Löwis" wrote: >>> libs/_sqlite3.lib 2K >> >> I think this is a summary of the entry points into one of the above >> DLLs for the benefit of other code wanting to link against it, but I'm >> not sure. > > Correct. I don't know why I include them in the MSI - they are there > because they were also shipped with the Wise installer. I see no > use - nobody should be linking against an extension module. They even cause trouble. Just yesterday I (well, not me: the pypy translation process) was caught by the presence of the "bz2.lib" file, which pypy found there, just because the linker lists c:\python25\LIBs before other directories. Of course the real bz2.lib, which defines the compression routines, was installed somewhere else, and compilation failed. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tracker cleanup - Roundup hacking report :)
"Daniel (ajax) Diniz" writes: > Feedback on meta-tracker open issues, as well as new RFEs and > replies to tracker-discuss threads should make the tracker meeting > your needs a bit more likely :) Thank you for http://issues.roundup-tracker.org/issue2550523>. I have wanted to participate on the Python issue tracker, but Roundup's lack of OpenID login support has made it more rewarding to simply discuss on mailing lists. I think that allowing people to log in using their *existing* identities will lower the barrier significantly to reporting bugs or contributing information. In short: +1 :-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
