[issue20584] On FreeBSD, signal.NSIG is smaller than biggest signal value
Steffen Daode Nurpmeso added the comment: Salut!, amis français! (Und auch sonst so, natürlich.) POSIX has recently standardized a NSIG_MAX constant in limits.h [1]: The value of {NSIG_MAX} shall be no greater than the number of signals that the sigset_t type (see [cross-ref to signal.h]) is capable of representing, ignoring any restrictions imposed by sigfillset() or sigaddset(). I'm personally following an advise of Rich Felker in the meantime: #ifdef NSIG_MAX # undef NSIG # define NSIG NSIG_MAX #elif !defined NSIG # define NSIG ((sizeof(sigset_t) * 8) - 1) #endif That is for old signals only, there; maybe reducing this to #undef NSIG #ifdef NSIG_MAX # define NSIG NSIG_MAX #else # define NSIG ((sizeof(sigset_t) * 8) - 1) #endif should do the trick for Python on any POSIX system? Ciao from, eh, gray, Germany :) [1] http://austingroupbugs.net/view.php?id=741#c1834 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20584 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11686] Update of some email/ __all__ lists
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: R. David Murray wrote [2012-03-17 03:51+0100]: Thanks for the patch, Steffen. Warm wink to the cleanroom-squatters! (I count this as the promised petit glass of red wine.) --steffen Forza Figa! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11686 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11935] MMDF/MBOX mailbox need utime
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Let me close this! I've just recently removed the real patch from my postman's next branch, because even that real implementation doesn't work reliable. I.e., please forget msg135791. It was true, but on the long run mutt(1) sometimes sees all, sometimes only some (really nuts), but most of the time it simply does not see just any box with new mail. That is, that plugged-in filesystem is simply handled as a pendant. Remarks: because that stdlib MBOX whispered Where Are Tho{u}, Brother to me all the {time}, i've done my own, also just recently: == postman: - test: 321 messages (5083760 bytes) [action=hunky-dory] = Dispatched 321 tickets to 1 box. [69853 refs] real 0m35.538s user 0m6.760s sys 0m0.904s .. = Dispatched 1963 tickets to 1 box. [93552 refs] real 0m38.860s user 0m8.697s sys 0m0.985s == stdlib: [83010 refs] real 1m3.862s user 0m10.151s sys 0m7.500s [93217 refs] real 7m24.958s user 2m0.174s sys 1m35.163s Was worth it. Have a good time! -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11935 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11686] Update of some email/ __all__ lists
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Closing this... -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11686 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11780] email.encoders are broken
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: I think this is historic either? As far as i remember you solved it as part of another issue... -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11780 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11701] email.parser.BytesParser().parse() closes file argument
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Right, and that is considered to be a non-issue due to that close() is allowed multiple times without causing harm. -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11701 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11466] getpass.getpass doesn't close tty file
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: -- nosy: -sdaoden ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11466 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12868] test_faulthandler.test_stack_overflow() failed on OpenBSD
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Heya. OpenBSD does support 1:1 threads via the RThread library since 2005, thanks to tedu@ and more fantastic guys! It about to be super-stable (one commit in 2011, last real fix in april 2010). (There is a techtalk from tedu@ (Ted Unangst) about this library on YouTube, btw.) -- nosy: +sdaoden ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12868 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12868] test_faulthandler.test_stack_overflow() failed on OpenBSD
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: In Python 3.3, you can use sys.thread_info to check which threading library is used. Great! I didn't know that! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12868 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12730] Python's casemapping functions are untrustworthy due to narrow/wide build issues
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: A sign! A sign! Someone with a name-name-name!! (Not a useful comment, i'm afraid.) -- nosy: +sdaoden ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12730 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12730] Python's casemapping functions are untrustworthy due to narrow/wide build issues
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: -- nosy: -sdaoden ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12730 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Are you willing to update your patch accordingly? I'm a vain rooster! I've used fullfsync() in 11877-standalone.1.diff! Sorry for that, haypo. :-/ 11877.fullsync-1.diff uses fullsync() and will instead always be provided when fsync() is available, to which it will fall back if no special operating system functionality is available. I really think this is the cleanest solution, because like this a user can state i want the strongest guarantees available on data integrity, and Python does just that. --Steffen Ciao, sdaoden(*)(gmail.com) ASCII ribbon campaign ( ) More nuclear fission plants against HTML e-mailXcan serve more coloured and proprietary attachments / \ and sounding animations -- Added file: http://bugs.python.org/file22759/11877.fullsync-1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -706,6 +706,31 @@ Availability: Unix, and Windows starting in 2.2.3. +.. function:: fullsync(fd) + + The POSIX standart requires that :c:func:`fsync` must transfer the buffered + data to the storage device, not that the data is actually written by the + device itself. It explicitely leaves it up to operating system implementors + whether users are given stronger guarantees on data integrity or not. Some + systems also offer special functions which overtake the part of making such + stronger guarantees, i.e., Mac OS X and NetBSD. :func:`fullsync` is + identical to :func:`fsync` unless there is such special functionality + available, in which case that will be used. + To strive for best-possible data integrity, the following can be done:: + + # Force writeout of local buffer modifications + f.flush() + # Then synchronize the changes to physical backing store + if hasattr(os, 'fsync'): + os.fullsync(f.fileno()) + + ..note:: + Calling this function may take a long time, since it may block + until the disk reports that the transfer has been completed. + + Availability: See :func:`fsync`. + + .. function:: ftruncate(fd, length) Truncate the file corresponding to file descriptor *fd*, so that it is at most diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -554,7 +554,7 @@ class TestInvalidFD(unittest.TestCase): singles = [fchdir, fdopen, dup, fdatasync, fstat, - fstatvfs, fsync, tcgetpgrp, ttyname] + fstatvfs, fsync, fullsync, tcgetpgrp, ttyname] #singles.append(close) #We omit close because it doesn'r raise an exception on some platforms def get_single(f): diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -1855,6 +1855,42 @@ { return posix_fildes(fdobj, fsync); } + +PyDoc_STRVAR(fullsync__doc__, +fullsync(fd)\n\n +force write of file buffers to disk, and the flush of disk caches\n +of the file given by file descriptor fd.); + +static PyObject * +fullsync(PyObject *self, PyObject *fdobj) +{ +/* See issue 11877 discussion */ +int res, fd = PyObject_AsFileDescriptor(fdobj); +if (fd 0) +return NULL; +if (!_PyVerify_fd(fd)) +return posix_error(); + +Py_BEGIN_ALLOW_THREADS +# if defined __APPLE__ +/* F_FULLFSYNC is not supported for all types of FDs/FSYSs; + * be on the safe side and test for inappropriate ioctl errors. + * Because plain fsync() may succeed even then, let it decide about error */ +res = fcntl(fd, F_FULLFSYNC); +if (res 0 errno == ENOTTY) +res = fsync(fd); +# elif defined __NetBSD__ +res = fsync_range(fd, FFILESYNC | FDISKSYNC, 0, 0); +# else +res = fsync(fd); +# endif +Py_END_ALLOW_THREADS + +if (res 0) +return posix_error(); +Py_INCREF(Py_None); +return Py_None; +} #endif /* HAVE_FSYNC */ #ifdef HAVE_FDATASYNC @@ -8953,6 +8989,7 @@ #endif #ifdef HAVE_FSYNC {fsync, posix_fsync, METH_O, posix_fsync__doc__}, +{fullsync,fullsync, METH_O, fullsync__doc__}, #endif #ifdef HAVE_FDATASYNC {fdatasync, posix_fdatasync, METH_O, posix_fdatasync__doc__}, ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Even PEP 3151 won't help. I don't understand. If the syscall supposed to flush the disk's buffer cache fails - be it fcntl() or sync_file_range() - I think the error should be propagated, not silently ignored and replaced with another syscall which doesn't have the same semantic. That's all. I'm with you theoretically - of course errors should be propagated to users so that they can act accordingly. And this is exactly what the patches do, *unless* it is an error which is not produced by the native fsync(2) call: -- 8 -- ?0%0[steffen@sherwood tmp]$ cat t.c #include errno.h #include fcntl.h #include stdio.h #include string.h #include unistd.h int main(void) { int r = fcntl(2, F_FULLFSYNC); fprintf(stderr, 1. %d: %d, %s\n, r, errno, strerror(errno)); errno = 0; r = fsync(2); fprintf(stderr, 2. %d: %d, %s\n, r, errno, strerror(errno)); return 0; } ?0%0[steffen@sherwood tmp]$ gcc -o t t.c ./t 1. -1: 25, Inappropriate ioctl for device 2. 0: 0, Unknown error: 0 ?0%0[steffen@sherwood tmp]$ grep -F 25 /usr/include/sys/errno.h #define ENOTTY 25 /* Inappropriate ioctl for device */ -- 8 -- So in fact the patches do what is necessary to make the changed version act just as the plain systemcall. - I favour haypos fullsync() approach Are you willing to update your patch accordingly? Both patches still apply onto the tip of friday noon: http://bugs.python.org/file22016/11877.9.diff, http://bugs.python.org/file22046/11877-standalone.1.diff. Again: i personally would favour os.fsync(fd, fullsync=True), because that is the only way to put reliability onto unaware facilities unaware (e.g. my S-Postman replaces os.fsync() with a special function so that reliability is injected in- and onto Python's mailbox.py, which calls plain os.fsync()), but i've been convinced that this is impossible to do. It seems to be impossible to change os.fsync() at all, because it has a standartized function prototype. So what do you mean? Shall i rewrite 11877-standalone.1.diff to always offer fullsync() whenever there is fsync()? This sounds to be a useful change, because testing hasattr() of the one would imply availability of the other. + Aaarrg! I'm a liar!! I lie about - data integrity!!! Well, actually, some hard disks lie about this too :-) Yeah. But hey: I feel save in New York City. I feel save in New York City. Nice weekend - and may the juice be with you! --Steffen Ciao, sdaoden(*)(gmail.com) ASCII ribbon campaign ( ) More nuclear fission plants against HTML e-mailXcan serve more coloured and proprietary attachments / \ and sounding animations -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Here is something unsorted and loose: - @neologix: One could argue that something had happened before the fsync(2), so that code which blindly did so is too dumb to do any right decision anyway. Even PEP 3151 won't help. - I favour haypos fullsync() approach, because that could probably make it into it. Yet Python doesn't offer any possibility to access NetBSD DISKSYNC stuff sofar. - Currently the programmer must be aware of any platform specific problems. I, for example, am not aware of Windows. How can i give any guarantee to users which (will) use my S-Postman on Windows? I need to put trust in turn into the Framework i am using - Python. And that makes me feel pretty breathless. - Fortunately Python is dynamic, so that one simply can replace os.fsync(). Works once only though (think signal handlers :=). + That is indeed the solution i'm using for my S-Postman, because *only* like this i can actually make Python's mailbox.py module reliable on Mac OS X! I can't give any guarantee for NetBSD, though i document it! + Aaarrg! I'm a liar!! I lie about - data integrity!!! --Steffen Ciao, sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: If Nir's analysis is right, and Antoines comment pushes me into this direction, (i personally have not looked at that code), then multiprocessing is completely brain-damaged and has been implemented by a moron. And yes, I know this is a bug tracker, and even that of Python. Nir should merge his last two messages into a single mail to python-dev, and those guys should give Nir or Thomas or a group of persons who have time and mental power a hg(1) repo clone and committer access to that and multiprocessing should be rewritten, maybe even from scratch, but i dunno. For the extremely unlikely case that all that doesn't happen maybe the patch of neologix should make it? --Steffen Ciao, sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Um, and just to add: i'm not watching out for anything, and it won't and it can't be me: ?0%0[steffen@sherwood sys]$ grep -F smp CHANGELOG.svn -B3 | grep -E '^r[[:digit:]]+' | tail -n 1 r162 | steffen | 2006-01-18 18:29:58 +0100 (Wed, 18 Jan 2006) | 35 lines --Steffen Ciao, sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: P.S.: I have to apologize, it's Tomaž, not Thomas. (And unless i'm mistaken this is pronounced TomAsch rather than the english Tommes, so i was just plain wrong.) --Steffen Ciao, sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: then multiprocessing is completely brain-damaged and has been implemented by a moron. Please do not use this kind of language. Being disrespectful to other people hurts the discussion. So i apologize once again. 'Still i think this should go to python-dev in the mentioned case. (BTW: there are religions without god, so whom shall e.g. i praise for the GIL?) --Steffen Ciao, sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: So sorry that i'm stressing this, hopefully it's the final message. Apples iterative kernel-update strategy resulted in these versions: 14:02 ~/tmp $ /usr/sbin/sysctl kern.version kern.version: Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 14:02 ~/tmp $ gcc -o zt osxversion.c -framework CoreServices 14:03 ~/tmp $ ./zt OS X version: 10.6.8 apple_osx_needs_fullsync: -1 I.e. the new patch uses 10.7.0 or =10.6.8 to avoid that FULLFSYNC disaster (even slower than the Macrohard memory allocator during Wintel partnership!), and we end up as: 14:03 ~/src/cpython $ ./python.exe -E -Wd -m test -r -w -uall test_mmap Using random seed 8466468 [1/1] test_mmap 1 test OK. P.S.: i still have no idea how to do '-framework CoreServices' regulary. Like i've said in #11046 i never used GNU Autoconf/M4, sorry. You know. Maybe the version check should be moved somewhere else and simply be exported, even replacing the stuff from platform.py? I don't know. Bye. -- Ciao, Steffen sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- Added file: http://bugs.python.org/file22593/11277.apple-fix-3.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst --- a/Doc/library/mmap.rst +++ b/Doc/library/mmap.rst @@ -88,7 +88,8 @@ To ensure validity of the created memory mapping the file specified by the descriptor *fileno* is internally automatically synchronized - with physical backing store on Mac OS X and OpenVMS. + with physical backing store on operating systems where this is + necessary, e.g. OpenVMS, and some buggy versions of Mac OS X. This example shows a simple way of using :class:`mmap`:: diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c --- a/Modules/mmapmodule.c +++ b/Modules/mmapmodule.c @@ -25,6 +25,8 @@ #define UNIX # ifdef __APPLE__ # include fcntl.h + +# include CoreServices/CoreServices.h # endif #endif @@ -65,6 +67,44 @@ #define my_getpagesize getpagesize #endif +# ifdef __APPLE__ +static void +apple_osx_needs_fullsync(long *use_fullsync) +{ +/* Issue #11277: mmap(2) bug with 32 bit sparse files. + * Apple fixed the bug before announcement of OS X Lion, but since we + * need to handle buggy versions, perform a once-only check to see if the + * running kernel requires the expensive sync. Fixed in 10.6.8, 10.7++. + * 0: F_FULLSYNC is required, 0: kernel has mmap(2) bug fixed */ +SInt32 ver; +*use_fullsync = 1; + +if (Gestalt(gestaltSystemVersion, ver) != noErr) +goto jleave; +/* SystemVersion(Major|Minor|BugFix) available at all? */ +if (ver 0x1040) +goto jleave; +if (Gestalt(gestaltSystemVersionMajor, ver) != noErr) +goto jleave; +if (ver 10) +goto jgood; +if (Gestalt(gestaltSystemVersionMinor, ver) != noErr) +goto jleave; +if (ver = 7) +goto jgood; +if (ver 6) +goto jleave; +if (Gestalt(gestaltSystemVersionBugFix, ver) != noErr) +goto jleave; +if (ver 8) +goto jleave; +jgood: +*use_fullsync = -1; +jleave: +return; +} +# endif /* __APPLE__ */ + #endif /* UNIX */ #include string.h @@ -1150,8 +1190,14 @@ #ifdef __APPLE__ /* Issue #11277: fsync(2) is not enough on OS X - a special, OS X specific fcntl(2) is necessary to force DISKSYNC and get around mmap(2) bug */ -if (fd != -1) -(void)fcntl(fd, F_FULLFSYNC); +if (fd != -1) { +/* (GIL protected) */ +static long use_fullsync /*= 0*/; +if (!use_fullsync) +apple_osx_needs_fullsync(use_fullsync); +if (use_fullsync 0) +(void)fcntl(fd, F_FULLFSYNC); +} #endif #ifdef HAVE_FSTAT # ifdef __VMS ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file22281/11277.apple-fix-2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: My suggestion to this would be that it should be outdated in the same way that Georg Brandl has suggested for changing the default encoding on python-dev [1], and add clear documentation on that, also in respect to the transition phase .. The problem is that multiprocessing itself, by construction, uses fork() with multiple threads. .. and maybe add some switches which allow usage of fork() for the time being. Today a '$ grep -Fir fork' does not show up threading.rst at all, which seems to be little in regard to the great problem. I would add a big fat note that multiprocessing.Process should be used instead today, because how about those of us who are not sophisticated enough to be appointed to standard committees? But anyway we should be lucky: fork(2) is UNIX specific, and thus it can be expected that all thread-safe libraries etc. are aware of the fact that they may be cloned by it. Except mine, of course. ,~) [1] http://mail.python.org/pipermail/python-dev/2011-June/112126.html -- Ciao, Steffen sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: How do you think multiprocessing.Process launches a new process? But it's a single piece of code under control and even multi-OS/multi-architecture test-coverage, not a general purpose Joe, you may just go that way and Python will handle it correctly? What i mean is: ten years ago (or so), Java did not offer true selection on sockets (unless i'm mistaken) - servers needed a 1:1 mapping of threads:sockets to handle connections?! But then, a this thread has finished the I/O, let's use it for something different seems to be pretty obvious. This is ok if it's your professor who is forcefully misleading you into the wrong direction, but otherwise you will have problems, maybe sooner, maybe later (, maybe never). And currently there is not a single piece of documentation which points you onto the problems. (And there *are* really people without Google.) The problem is that it looks so simple and easy - but it's not. In my eyes it's an unsolvable problem. And for the sake of resource usage, simplicity and execution speed i appreciate all solutions which don't try to do the impossible. I want to add that all this does not really help just as long just *any* facility which is used by Python *itself* is not under control of atfork. Solaris e.g. uses atfork for it's memory allocator, because that is surely needed if anything else but async-safe facilities are used in the newly forked process. Can Python give that guarantee for all POSIX systems it supports? Good night. -- Ciao, Steffen sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11728] mbox parser incorrect behaviour
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Hello Valery Masiutsin, i recently stumbled over this while searching for the link to the standart i've stored in another issue. (Without being logged in, say.) The de-facto standart (http://qmail.org/man/man5/mbox.html) says: HOW A MESSAGE IS READ A reader scans through an mbox file looking for From_ lines. Any From_ line marks the beginning of a message. The reader should not attempt to take advantage of the fact that every From_ line (past the beginning of the file) is preceded by a blank line. This is however the recent version. The mbox manpage of my up-to-date Mac OS X 10.6.7 does not state this, for example. It's from 2002. However, all known MBOX standarts, i.e. MBOXO, MBOXRD, MBOXCL, require proper quoting of non-From_ From lines (by preceeding with ''). So your example should not fail in Python. (But hey - are you sure *that* has been produced by Perl?) You're right however that Python seems to only support the old MBOXO way of un-escaping only plain From to/from From , which is not even mentioned anymore in the current standart - that only describes MBOXRD ((*From ) - +match.group(1)). (Lucky me: i own Mac OS X, otherwise i wouldn't even know.) Thus you're in trouble if the unescaping is performed before the split.. This is another issue, though: MBOX parser uses MBOXO algorithm. ; - Ciao, Steffen -- nosy: +sdaoden ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11728 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: @ Ronald Oussoren wrote: if major = 10: # We're on OSX 10.6 or earlier enableWorkaround() (You sound as if you participate in an interesting audiophonic event. 27 imac's are indeed great recording studio hardware. But no Coffee Shops in California - br.) -- Ciao, Steffen sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Ok, this patch could be used. *Unless* the code is not protected by the GIL. - Gestalt usage is a bit more complicated according to http://www.cocoadev.com/index.pl?DeterminingOSVersion unless Python only supports OS X 10.4 and later. (And platform.py correctly states that in _mac_ver_gestalt(), but see below.) - Due to usage of Gestalt, '-framework CoreServices' must be linked against mmapmodule.c. The Python configuration stuff is interesting for me, i managed compilation by adding the line mmap mmapmodule.c -framework CoreServices to Modules/Setup, but i guess it's only OS X which is happy about that. platform.py: _mac_ver_xml() should be dropped entirely according to one of Ned Deily's links (never officially supported), and _mac_ver_gestalt() obviously never executed because i guess it would fail due to versioninfo. Unless i missed something. By the way: where do you get the info from? sys1, sys2, sys3? Cannot find it anywhere, only the long names, e.g. gestaltSystemVersionXy. Note that i've mailed Apple. I did not pay 99$ or even 249$, so i don't know if there will be a response. -- Ciao, Steffen sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- Added file: http://bugs.python.org/file22281/11277.apple-fix-2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst --- a/Doc/library/mmap.rst +++ b/Doc/library/mmap.rst @@ -88,7 +88,8 @@ To ensure validity of the created memory mapping the file specified by the descriptor *fileno* is internally automatically synchronized - with physical backing store on Mac OS X and OpenVMS. + with physical backing store on operating systems where this is + necessary, e.g. OpenVMS, and some buggy versions of Mac OS X. This example shows a simple way of using :class:`mmap`:: diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c --- a/Modules/mmapmodule.c +++ b/Modules/mmapmodule.c @@ -25,6 +25,8 @@ #define UNIX # ifdef __APPLE__ # include fcntl.h + +# include CoreServices/CoreServices.h # endif #endif @@ -65,6 +67,39 @@ #define my_getpagesize getpagesize #endif +# ifdef __APPLE__ +static void +apple_osx_needs_fullsync(long *use_fullsync) +{ +/* Issue #11277: mmap(2) bug with 32 bit sparse files. + * Apple fixed the bug before announcement of OS X Lion, but since we + * need to handle buggy versions, perform a once-only check to see if the + * running kernel requires the expensive sync. + * 0: F_FULLSYNC is required, 0: kernel has mmap(2) bug fixed */ +SInt32 ver; + +*use_fullsync = 1; +if (Gestalt(gestaltSystemVersion, ver) != noErr ver = 0x1040) { +if (Gestalt(gestaltSystemVersionMajor, ver) != noErr) +goto jleave; +if (ver 10) { +*use_fullsync = -1; +goto jleave; +} + +if (Gestalt(gestaltSystemVersionMinor, ver) != noErr) +goto jleave; +if (ver = 7) { +*use_fullsync = -1; +goto jleave; +} +} + +jleave: +return; +} +# endif /* __APPLE__ */ + #endif /* UNIX */ #include string.h @@ -1128,8 +1163,14 @@ #ifdef __APPLE__ /* Issue #11277: fsync(2) is not enough on OS X - a special, OS X specific fcntl(2) is necessary to force DISKSYNC and get around mmap(2) bug */ -if (fd != -1) -(void)fcntl(fd, F_FULLFSYNC); +if (fd != -1) { +/* (GIL protected) */ +static long use_fullsync /*= 0*/; +if (!use_fullsync) +apple_osx_needs_fullsync(use_fullsync); +if (use_fullsync 0) +(void)fcntl(fd, F_FULLFSYNC); +} #endif #ifdef HAVE_FSTAT # ifdef __VMS ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file22273/11277.apple-fix.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Aehm, note that Apple has fixed the mmap(2) bug!! I'm still surprised and can't really believe it, but it's true! Just in case you're interested, i'll apply an updated patch. Maybe Ned Deily should have a look at the version check, which does not apply yet, but i don't know any other way to perform exact version checking. (Using 10.6.7 is not enough, it must be 10.7.0; uname -a yet reports that all through, but no CPP symbol does!?) -- Added file: http://bugs.python.org/file22273/11277.apple-fix.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: @ Ned Deily rep...@bugs.python.org wrote (2011-06-07 19:43+0200): Thanks for the update. Since the fix will be in a future version of OS X 10.7 Lion, and which has not been released yet, so it is not appropriate to change mmap until there has been an opportunity to test it. It's really working fine. That i see that day! (Not that they start to fix the CoreAudio crashes...) But even then, we would need to be careful about adding a compile-time test as OS X binaries are often built to be compatible for a range of operating system version so avoid adding compilation conditionals unless really necessary. If after 10.7 is released and someone is able to test that it works as expected, the standard way to support it would be to use the Apple-supplied availability macros to test for the minimum supported OS level of the build assuming it makes enough of a performance difference to bother to do so Of course it only moves the delay from before mmap(2) to after close(2). Well, i don't know, if hardcoding is not an option, a dynamic sysctl(2) lookup may do: kern.version = Darwin Kernel Version 10.7.0: Sat Jan 29 15:17:16 PST 2011 This is obviously not the right one. :) -- Ciao, Steffen sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11700] mailbox.py proxy updates
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Hello, David, i'm about to remind you that this issue is still open (and the next release is about to come soon). I think we do agree at least in the fact that this is a bug :). Well, your mailbox_close_twice.patch no. 2 still imports on the current tip. (I'm still a bit disappointed that you don't want to -a-r-m- upgrade the proxies to the full implementation i've posted. But it's ok. By the way: you're the first american i know who doesn't want to upgrade his arms! And i do have had an ex-uncle who is a fellow countryman of yours.) Regards from Germany during kitschy pink sunset -- Added file: http://bugs.python.org/file22095/11700.yeah-review.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11700 ___diff --git a/Lib/mailbox.py b/Lib/mailbox.py --- a/Lib/mailbox.py +++ b/Lib/mailbox.py @@ -1864,97 +1864,142 @@ Message with MMDF-specific properties. -class _ProxyFile: -A read-only wrapper of a file. +class _ProxyFile(io.BufferedIOBase): +A io.BufferedIOBase inheriting read-only wrapper for a seekable file. +It supports __iter__() and the context-manager protocol. + +def __init__(self, file, pos=None): +io.BufferedIOBase.__init__(self) +self._file = file +self._pos = file.tell() if pos is None else pos +self._close = True +self._is_open = True -def __init__(self, f, pos=None): -Initialize a _ProxyFile. -self._file = f -if pos is None: -self._pos = f.tell() +def _set_noclose(self): +Subclass hook - use to avoid closing internal file object. +self._close = False + +def _closed_check(self): +Raise ValueError if not open. +if not self._is_open: +raise ValueError('I/O operation on closed file') + +def close(self): +if self._close: +self._close = False +self._file.close() +del self._file +self._is_open = False + +@property +def closed(self): +return not self._is_open + +def flush(self): +# Not possible because it gets falsely called (issue 11700) +#raise io.UnsupportedOperation('flush') +pass + +def _read(self, size, read_method, readinto_arg=None): +if size is None or size 0: +size = -1 +self._file.seek(self._pos) +if not readinto_arg: +result = read_method(size) else: -self._pos = pos +result = read_method(readinto_arg) +if result len(readinto_arg): +del readinto_arg[result:] +self._pos = self._file.tell() +return result -def read(self, size=None): -Read bytes. +def readable(self): +self._closed_check() +return True + +def read(self, size=-1): +self._closed_check() +if size is None or size 0: +return self.readall() return self._read(size, self._file.read) -def read1(self, size=None): -Read bytes. +def read1(self, size=-1): +self._closed_check() +if size is None or size 0: +return b'' return self._read(size, self._file.read1) -def readline(self, size=None): -Read a line. +def readinto(self, by_arr): +self._closed_check() +return self._read(len(by_arr), self._file.readinto, by_arr) + +def readall(self): +self._closed_check() +self._file.seek(self._pos) +if hasattr(self._file, 'readall'): +result = self._file.readall() +else: +dl = [] +while 1: +i = self._file.read(8192) +if len(i) == 0: +break +dl.append(i) +result = b''.join(dl) +self._pos = self._file.tell() +return result + +def readline(self, size=-1): +self._closed_check() return self._read(size, self._file.readline) -def readlines(self, sizehint=None): -Read multiple lines. +def readlines(self, sizehint=-1): result = [] for line in self: result.append(line) -if sizehint is not None: +if sizehint = 0: sizehint -= len(line) if sizehint = 0: break return result +def seekable(self): +self._closed_check() +return True + +def seek(self, offset, whence=0): +self._closed_check() +if whence == 1: +self._file.seek(self._pos) +self._pos = self._file.seek(offset, whence) +return self._pos + +def tell(self): +self._closed_check() +return self._pos + +def writable(self): +self._closed_check() +return False
[issue11700] mailbox.py proxy updates
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file22095/11700.yeah-review.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11700 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12102] mmap requires file to be synced
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: that doesn't make me any good Well - 'can only be better than myself, so i'll take that as yes :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12102 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12102] mmap requires file to be synced
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file22020/12102.1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12102 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -810,6 +810,35 @@ Availability: Unix, and Windows. +.. function:: fullfsync(fd) + + The POSIX standart requires that :c:func:`fsync` must transfer the buffered + data to the storage device, not that the data is actually written by the + device itself. It explicitely leaves it up to operating system implementors + whether users are given stronger guarantees on data integrity or not. Some + systems also offer special functions which overtake the part of making such + stronger guarantees, i.e., Mac OS X and NetBSD. + + This non-standart function is *optionally* made available to access such + special functionality when feasible. It will force write of file buffers to + disk and the flush of disk caches of the file given by file descriptor *fd*. + To strive for best-possible data integrity, the following can be done:: + + # Force writeout of local buffer modifications + f.flush() + # Then synchronize the changes to physical backing store + if hasattr(os, 'fullfsync'): + os.fullfsync(f.fileno()) + elif hasattr(os, 'fsync'): + os.fsync(f.fileno()) + + ..note:: + Calling this function may take a long time, since it may block + until the disk reports that the transfer has been completed. + + Availability: Unix. + + .. function:: ftruncate(fd, length) Truncate the file corresponding to file descriptor *fd*, so that it is at most diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -835,12 +835,12 @@ class TestInvalidFD(unittest.TestCase): singles = [fchdir, dup, fdopen, fdatasync, fstat, - fstatvfs, fsync, tcgetpgrp, ttyname] + fstatvfs, fsync, fullfsync, tcgetpgrp, ttyname] #singles.append(close) -#We omit close because it doesn'r raise an exception on some platforms +# We omit close because it doesn't raise an exception on some platforms def get_single(f): def helper(self): -if hasattr(os, f): +if hasattr(os, f): self.check(getattr(os, f)) return helper for f in singles: diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -174,6 +174,11 @@ #endif /* ! __IBMC__ */ #ifndef _MSC_VER + /* os.fullfsync()? */ +# if (defined HAVE_FSYNC ((defined __APPLE__ defined F_FULLFSYNC) || \ + (defined __NetBSD__ defined FDISKSYNC))) +# define PROVIDE_FULLFSYNC +# endif #if defined(__sgi)_COMPILER_VERSION=700 /* declare ctermid_r if compiling with MIPSPro 7.x in ANSI C mode @@ -2129,6 +2134,41 @@ { return posix_fildes(fdobj, fsync); } + +# ifdef PROVIDE_FULLFSYNC +PyDoc_STRVAR(fullfsync__doc__, +fullfsync(fd)\n\n +force write of file buffers to disk, and the flush of disk caches\n +of the file given by file descriptor fd.); + +static PyObject * +fullfsync(PyObject *self, PyObject *fdobj) +{ +/* See issue 11877 discussion */ +int res, fd = PyObject_AsFileDescriptor(fdobj); +if (fd 0) +return NULL; +if (!_PyVerify_fd(fd)) +return posix_error(); + +Py_BEGIN_ALLOW_THREADS +# if defined __APPLE__ +/* F_FULLFSYNC is not supported for all types of FDs/FSYSs; + * be on the safe side and test for inappropriate ioctl errors */ +res = fcntl(fd, F_FULLFSYNC); +if (res 0 errno == ENOTTY) +res = fsync(fd); +# elif defined __NetBSD__ +res = fsync_range(fd, FFILESYNC | FDISKSYNC, 0, 0); +# endif +Py_END_ALLOW_THREADS + +if (res 0) +return posix_error(); +Py_INCREF(Py_None); +return Py_None; +} +# endif /* PROVIDE_FULLFSYNC */ #endif /* HAVE_FSYNC */ #ifdef HAVE_SYNC @@ -9473,6 +9513,9 @@ #endif #ifdef HAVE_FSYNC {fsync, posix_fsync, METH_O, posix_fsync__doc__}, +# ifdef PROVIDE_FULLFSYNC +{fullfsync, fullfsync, METH_O, fullfsync__doc__}, +# endif #endif #ifdef HAVE_SYNC {sync,posix_sync, METH_NOARGS, posix_sync__doc__}, -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: (This was an attachment to an empty mail message.) -- Added file: http://bugs.python.org/file22046/11877-standalone.1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12102] mmap requires file to be synced
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Looked at it again and i think it's much better english with an additional ..to ensure that local... @Ross, aren't you a native english speaker? What do you say? -- Added file: http://bugs.python.org/file22048/12102.2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12102 ___diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst --- a/Doc/library/mmap.rst +++ b/Doc/library/mmap.rst @@ -21,6 +21,11 @@ :func:`os.open` function, which returns a file descriptor directly (the file still needs to be closed when done). +..note:: + If you want to create a memory-mapping for a writable, buffered file, you + should :func:`flush` the file first. This is necessary to ensure that local + modifications to the buffers are actually available to the mapping. + For both the Unix and Windows versions of the constructor, *access* may be specified as an optional keyword parameter. *access* accepts one of three values: :const:`ACCESS_READ`, :const:`ACCESS_WRITE`, or :const:`ACCESS_COPY` ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Excusing myself seems to be the only probates Mittel. @Antoine Pitrou: It was a real shame to read your mail. (It's sometimes so loud that i don't even hear what i write.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12102] mmap requires file to be synced
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: @ rion wrote (2011-05-18 12:39+0200): just document it or fix. Hello, zion, Victor, i'm proposing a documentation patch. It applies to 2.7 and 3.3 (from yesterday). -- keywords: +patch Added file: http://bugs.python.org/file22020/12102.1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12102 ___diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst --- a/Doc/library/mmap.rst +++ b/Doc/library/mmap.rst @@ -21,6 +21,11 @@ :func:`os.open` function, which returns a file descriptor directly (the file still needs to be closed when done). +..note:: + If you want to create a memory-mapping for a writable, buffered file, you + should :func:`flush` the file first. This is necessary to ensure local + modifications to the buffers are actually available to the mapping. + For both the Unix and Windows versions of the constructor, *access* may be specified as an optional keyword parameter. *access* accepts one of three values: :const:`ACCESS_READ`, :const:`ACCESS_WRITE`, or :const:`ACCESS_COPY` ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12102] mmap requires file to be synced
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: @ STINNER Victor wrote (2011-05-18 14:33+0200): I don't think that Python should guess what the user expects (i.e. Python should not sync the file *implicitly*). Before i've found F_FULLFSYNC i indeed have had a solution which tracked the open() of all files, so that mmap() could check wether a file had been opened in write mode or not. ;| Python does not do that by default (AFAIK). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12102 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: @ Nir Aides wrote (2011-05-16 20:57+0200): Steffen, can you explain in layman's terms? I am the layman here. Charles-François has written a patch for Python which contradicted his own proposal from msg135079, but he seems to have tested a lot so that he then was even able to prove that his own proposal was correct. His new patch does implement that with a nice introductional note. He has also noticed that the only really safe solution is to simply disallow multi-threading in programs which fork(). And this either-or is exactly the conclusion we have taken and implemented in our C++ library - which is not an embeddable programming language that needs to integrate nicely in whatever environment it is thrown into, but even replaces main(). And i don't know any application which cannot be implemented regardless of fork()-or-threads instead of fork()-and-threads. (You *can* have fork()+exec()-and-threads at any time!) So what i tried to say is that it is extremely error-prone and resource intensive to try to implement anything that tries to achieve both. I.e. on Solaris they do have a forkall() and it seems they have atfork handlers for everything (and even document that in the system manual). atfork handlers for everything!! And for what? To implement a standart which is obviously brain-dead because it is *impossible* to handle it - as your link has shown this is even confessed by members of the committee. And writing memory in the child causes page-faults. That's all i wanted to say. (Writing this mail required more than 20 minutes, the mentioned one was out in less than one. And it is much more meaningful AFAIK.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Thank you, thank you, thank you. I'm a bit irritated that a french man treats a wet-her as a typo! What if *i* like it?? In fact it is a fantastic physical backing store. Unbeatable. Well and about dropping the fsync() in case the fcntl() fails with ENOTTY. This is Esc2dd, which shouldn't hurt a committer. I'm convinced that full_fsync=False is optional and false by default, but i don't trust Apple. I've seen a reference to an atomic file somewhere on bugs.python.org and that does fsync() first followed by fcntl() if FULLFSYNC is available. Thus, if someone knows about that, she may do so, but otherwise i would guess he doesn't, and in that case i would not expect ENOTTY from an fsync() - still i want a full flush! This is what NetBSD describes: NOTES For optimal efficiency, the fsync_range() call requires that the file system containing the file referenced by fd support partial synchronization of file data. For file systems which do not support partial synchronization, the entire file will be synchronized and the call will be the equivalent of calling fsync(). But Apple is *s*spcil* again. Happy birthday. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: I've dropped wet-her! I hope now you're satisfied! So the buffer cache is all which remains hot. How deserted! And you could also add a test (I guess that just calling fsync with full_sync=True on a valid FD would be enough. I was able to add two tests as an extension to what is yet tested about os.fsync(), but that uses an invalid fd. (At least it enters the conditional and fails as expected.) I'm not sure static is necessary, I'd rather make it const. Yes.. This code is correct as it is, see other extension modules in the stdlib for other examples of this pattern ..but i've used copy+paste here. And you could also add a test (I guess that just calling fsync with full_sync=True on a valid FD would be enough. The alternative would be that full_sync Ok, i've renamed full_fsync to full_sync. -- Added file: http://bugs.python.org/file22016/11877.9.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -798,7 +798,7 @@ Availability: Unix. -.. function:: fsync(fd) +.. function:: fsync(fd, full_sync=False) Force write of file with filedescriptor *fd* to disk. On Unix, this calls the native :c:func:`fsync` function; on Windows, the MS :c:func:`_commit` function. @@ -807,6 +807,15 @@ ``f.flush()``, and then do ``os.fsync(f.fileno())``, to ensure that all internal buffers associated with *f* are written to disk. + The POSIX standart requires that :c:func:`fsync` must transfer the buffered + data to the storage device, not that the data is actually written by the + device itself. It explicitely leaves it up to operating system implementors + whether users are given stronger guarantees on data integrity or not. Some + systems also offer special functions which overtake the part of making such + stronger guarantees, i.e., Mac OS X and NetBSD. The optional *full_sync* + argument can be used to enforce usage of these special functions if that is + appropriate for the *fd* in question. + Availability: Unix, and Windows. diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -837,10 +837,10 @@ singles = [fchdir, dup, fdopen, fdatasync, fstat, fstatvfs, fsync, tcgetpgrp, ttyname] #singles.append(close) -#We omit close because it doesn'r raise an exception on some platforms +# We omit close because it doesn't raise an exception on some platforms def get_single(f): def helper(self): -if hasattr(os, f): +if hasattr(os, f): self.check(getattr(os, f)) return helper for f in singles: @@ -855,6 +855,11 @@ self.fail(%r didn't raise a OSError with a bad file descriptor % f) +def test_fsync_arg(self): +if hasattr(os, fsync): +self.check(os.fsync, True) +self.check(os.fsync, False) + def test_isatty(self): if hasattr(os, isatty): self.assertEqual(os.isatty(support.make_bad_fd()), False) diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -2121,13 +2121,50 @@ #ifdef HAVE_FSYNC PyDoc_STRVAR(posix_fsync__doc__, -fsync(fildes)\n\n\ -force write of file with filedescriptor to disk.); - -static PyObject * -posix_fsync(PyObject *self, PyObject *fdobj) -{ -return posix_fildes(fdobj, fsync); +fsync(fildes, full_sync=False)\n\n +force write of file buffers with fildes to disk;\n +full_sync forces flush of disk caches in case fsync() alone is not enough.); + +static PyObject * +posix_fsync(PyObject *self, PyObject *args, PyObject *kwargs) +{ +PyObject *fdobj; +int full_sync = 0; +static char *keywords[] = {fd, full_sync, NULL }; + +if (!PyArg_ParseTupleAndKeywords(args, kwargs, O|i, keywords, + fdobj, full_sync)) +return NULL; + +/* See issue 11877 discussion */ +# if ((defined __APPLE__ defined F_FULLFSYNC) || \ + (defined __NetBSD__ defined FDISKSYNC)) +if (full_sync != 0) { +int res, fd = PyObject_AsFileDescriptor(fdobj); +if (fd 0) +return NULL; +if (!_PyVerify_fd(fd)) +return posix_error(); + +Py_BEGIN_ALLOW_THREADS +# if defined __APPLE__ +/* F_FULLFSYNC is not supported for all types of FDs/FSYSs; + * be on the safe side and test for inappropriate ioctl errors */ +res = fcntl(fd, F_FULLFSYNC); +if (res 0 errno == ENOTTY) +res = fsync(fd); +# elif defined __NetBSD__ +res = fsync_range(fd, FFILESYNC|FDISKSYNC, 0, 0); +# endif +Py_END_ALLOW_THREADS
[issue11877] Change os.fsync() to support physical backing store syncs
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file21986/11877.8.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Finally, depending on the workload, it could have a significant performance impact. Oh yes (first replaces os.fsync() with an in-python wrapper): 18:12 ~/tmp $ ll mail ls: mail: No such file or directory 18:12 ~/tmp $ ll X-MAIL 312 -rw-r- 1 steffen staff 315963 15 May 17:49 X-MAIL 18:12 ~/tmp $ time s-postman.py --folder=~/tmp/mail --dispatch=X-MAIL Dispatched 37 tickets to 4 targets. real0m4.638s user0m0.974s sys 0m0.160s 18:13 ~/tmp $ rm -rf mail 18:13 ~/tmp $ time s-postman.py --folder=~/tmp/mail --dispatch=X-MAIL Dispatched 37 tickets to 4 targets. real0m1.228s user0m0.976s sys 0m0.122s (I'm using the first one.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: @ Charles-François Natali wrote (2011-05-15 01:14+0200): So if we really wanted to be safe, the only solution would be to forbid fork() in a multi-threaded program. Since it's not really a reasonable option But now - why this? The only really acceptable thing if you have control about what you are doing is the following: class SMP::Process /*! * \brief Daemonize process. *[.] * \note * The implementation of this function is not trivial. * To avoid portability no-goes and other such problems, * you may \e not call this function after you have initialized * Thread::enableSMP(), * nor may there (have) be(en) Child objects, * nor may you have used an EventLoop! * I.e., the process has to be a single threaded, synchronous one. * [.] */ pub static si32 daemonize(ui32 _daemon_flags=df_default); namespace SMP::POSIX /*! * \brief \fn fork(2). *[.] * Be aware that this passes by all \SMP and Child related code, * i.e., this simply \e is the system-call. * Signal::resetAllSignalStates() and Child::killAll() are thus if * particular interest; thread handling is still entirely up to you. */ pub static sir fork(void); Which kind of programs cannot be written with this restriction? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: @ Charles-François Natali rep...@bugs.python.org wrote (2011-05-13 13:24+0200): I happily posted a reinit patch I must say in advance that we have implemented our own thread support 2003-2005 and i'm thus lucky not to need to use anything else ever since. So. And of course i have no overview about Python. But i looked and saw no errors in the default path and the tests run without errors. Then i started to try your semaphore path which is a bit problematic because Mac OS X doesn't offer anon sems ;). ( By the way, in PyThread_acquire_lock_timed() these lines if (microseconds 0) MICROSECONDS_TO_TIMESPEC(microseconds, ts); result in these compiler warnings. python/thread_pthread.h: In function ‘PyThread_acquire_lock_timed’: Python/thread_pthread.h:424: warning: ‘ts.tv_sec’ may be used uninitialized in this function Python/thread_pthread.h:424: warning: ‘ts.tv_nsec’ may be used uninitialized in this function ) #ifdef USE_SEMAPHORES #define broken_sem_init broken_sem_init static int broken_sem_init(sem_t **sem, int shared, unsigned int value) { int ret; auto char buffer[32]; static long counter = 3000; sprintf(buffer, %016ld, ++counter); *sem = sem_open(buffer, O_CREAT, (mode_t)0600, (unsigned int)value); ret = (*sem == SEM_FAILED) ? -1 : 0; //printf(BROKEN_SEM_INIT WILL RETURN %d (value=%u)\n, ret,value); return ret; } static int sem_timedwait(sem_t *sem, struct timespec *ts) { int success = -1, iters = 1000; struct timespec now, wait; printf(STARTING LOOP\n); for (;;) { if (sem_trywait(sem) == 0) { printf(TRYWAIT OK\n); success = 0; break; } wait.tv_sec = 0, wait.tv_nsec = 200 * 1000; //printf(DOWN ); fflush(stdout); nanosleep(wait, NULL); MICROSECONDS_TO_TIMESPEC(0, now); //printf(WOKE UP NOW=%ld:%ld END=%ld:%ld\n, now.tv_sec,now.tv_nsec, ts-tv_sec,ts-tv_nsec); if (now.tv_sec ts-tv_sec || (now.tv_sec == ts-tv_sec now.tv_nsec = ts-tv_nsec)) break; if (--iters 0) { printf(BREAKING OFF LOOP, 1000 iterations\n); errno = ETIMEDOUT; break; } } return success; } #define sem_destroy sem_close typedef struct _pthread_lock { sem_t *sem; struct _pthread_lock*next; sem_t sem_buf; } pthread_lock; #endif plus all the changes the struct change implies, say. Yes it's silly, but i wanted to test. And this is the result: == CPython 3.3a0 (default:804abc2c60de+, May 14 2011, 01:09:53) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] == Darwin-10.7.0-i386-64bit little-endian == /Users/steffen/src/cpython/build/test_python_19230 Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=1, verbose=0, bytes_warning=0, quiet=0) Using random seed 1362049 [1/1] test_threading STARTING LOOP test_acquire_contended (test.test_threading.LockTests) ... ok test_acquire_destroy (test.test_threading.LockTests) ... ok test_acquire_release (test.test_threading.LockTests) ... ok test_constructor (test.test_threading.LockTests) ... ok test_different_thread (test.test_threading.LockTests) ... ok test_reacquire (test.test_threading.LockTests) ... ok test_state_after_timeout (test.test_threading.LockTests) ... ok test_thread_leak (test.test_threading.LockTests) ... ok test_timeout (test.test_threading.LockTests) ... STARTING LOOP TRYWAIT OK FAIL test_try_acquire (test.test_threading.LockTests) ... ok test_try_acquire_contended (test.test_threading.LockTests) ... ok test_with (test.test_threading.LockTests) ... ok test__is_owned (test.test_threading.PyRLockTests) ... ok test_acquire_contended (test.test_threading.PyRLockTests) ... ok test_acquire_destroy (test.test_threading.PyRLockTests) ... ok test_acquire_release (test.test_threading.PyRLockTests) ... ok test_constructor (test.test_threading.PyRLockTests) ... ok test_different_thread (test.test_threading.PyRLockTests) ... ok test_reacquire (test.test_threading.PyRLockTests) ... ok test_release_unacquired (test.test_threading.PyRLockTests) ... ok test_thread_leak (test.test_threading.PyRLockTests) ... ok test_timeout (test.test_threading.PyRLockTests) ... STARTING LOOP TRYWAIT OK FAIL test_try_acquire (test.test_threading.PyRLockTests) ... ok test_try_acquire_contended (test.test_threading.PyRLockTests) ... ok test_with (test.test_threading.PyRLockTests) ... ok test__is_owned (test.test_threading.CRLockTests) ... ok test_acquire_contended (test.test_threading.CRLockTests) ... ok test_acquire_destroy (test.test_threading.CRLockTests) ... ok test_acquire_release (test.test_threading.CRLockTests) ... ok test_constructor (test.test_threading.CRLockTests) ... ok test_different_thread (test.test_threading.CRLockTests) ... ok test_reacquire
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Just adding more notes on that by reactivating one of haypo's links from #8604. (And: maybe some Linux documentation should be updated?) From Theodore Ts'o, http://www.linuxfoundation.org/news-media/blogs/browse/2009/03/don’t-fear-fsync: As the Eat My Data presentation points out very clearly, the only safe way according that POSIX allows for requesting data written to a particular file descriptor be safely stored on stable storage is via the fsync() call. Linux’s close(2) man page makes this point very clearly: A successful close does not guarantee that the data has been successfully saved to disk, as the kernel defers writes. It is not common for a file system to flush the buffers when the stream is closed. If you need to be sure that the data is physically stored use fsync(2). Why don’t application programmers follow these sage words? These three reasons are most often given as excuses: - (Perceived) performance problems with fsync() - The application only needs atomicity, but not durability - The fsync() causing the hard drive to spin up unnecessarily in laptop_mode (Don't ask me why i'm adding this note though. I should have searched for it once i've opened that issue? Bah!!! Ts'o did not write that article for me. He'd better hacked.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12060] Python doesn't support real time signals
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Dunno. The patch is not completely safe. Yeah it will not work without atomic ops. Unfortunately the C standart seems to go into a direction noone understands - as if a atomic_compare_and_swap() would not suffice! Do you know any machine language which reflects what that standart draft describes? I don't. The NSIG detection of Modules/signalmodule.c uses 64 as a fallback. 32 seems to be more reasonable. And you test against it instead of RTMAX in the patch. (signalmodule.c also exports Python constants RTMIN and RTMAX even though the standart explicitely allows these values to be non-constants; http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/signal.h.html; last time i've done anything on signals - in 2005 - that was used nowhere - Linux, FreeBSD - though.) Often there is a huge whole in between NSIG and RTMIN, but struct Handlers is 8 or 12 bytes (unless the compiler does the alignment - ouuh), so 32 unused members in Handlers[] will not cost the world anyway; on Mac OS X (no RTSIG support?!? ;) Python is at least 6 megabytes of memory anyway. And does anyone actually know why the last time i looked after this (on Linux, then) realtime signals had a default action EQ SIGABRT? Armchair crouchers... -- nosy: +sdaoden ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12060 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12060] Python doesn't support real time signals
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: On my Linux box, Python 3.3 says that signal.NSIG is equal to 65 which looks correct. On FreeBSD NSIG only counts old signals (32, one 32 bit mask), SIGRTMIN is 65 and SIGRTMAX is 126. Our internal old signal.h states * If we do have realtime signals, #rtmin is 35 (i.e., * #nsig, FreeBSD+) or something like 38 or even 40 (Linux), * and #rtmax is most likely 64 (Linux) or 128 (FreeBSD+). so that this seems to be somewhat constant in time. (#rtmin: we take some of those RT sigs for internal purposes if possible. This was maybe a bad and expensive design decision.) Why do you care about the default action? * \brief Hooking program crashes (\psa crash.h crash.h\epsa). * \note * Installed hooks (normally) execute from within an internal * signal handler! So many syscalls for things which don't matter almost ever. And that may even cost context-switches sometimes. I don't understand: I don't use RTMAX in my patch. +for (signum = 1; signum NSIG; signum++) { This will not catch the extended signal range on FreeBSD. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12060 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: [.] OSError: [Errno 22] Invalid argument Sorry, i didn't know that. Mac OS X (2.5 and 2.6 Apple shipped): 21:43 ~/tmp $ python2.5 -c 'import os; os.fsync(1)'; echo $? 0 21:43 ~/tmp $ python2.6 -c 'import os; os.fsync(1)'; echo $? 0 21:43 ~/tmp $ python2.7 -c 'import os; os.fsync(1)'; echo $? 0 21:43 ~/tmp $ python3 -c 'import os; os.fsync(1)'; echo $? 0 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: So I think you should stick with the previous version (well, if the full sync fails on other FDs, then it's another story, but in that case it should just be dropped altogether if it's not reliable...). Strong stuff. *This* is the version which should have been implemented from the beginning, but Apple states F_FULLFSYNC Does the same thing as fsync(2) then asks the drive to flush all buffered data to the permanent storage device (arg is ignored). This is currently implemented on HFS, MS-DOS (FAT), and Universal Disk Format (UDF) file systems. and i thought - fsync (maybe move buffers to Queue; do reorder Queue as approbiate) - do call fsys impl. to .. whatever That's why i had a version of the patch which did 'fsync();fcntl();' because it would have been an additional syscall but the fsync() part would possibly have been essentially skipped over ..unless.. Linux RC scripts had 'sync sync sync' but it does not seem to be necessary any more (was it ever - i don't know). But who knows if that fcntl will fail on some non-noted fsys? I think it's better to be on the safe side. Quoting you, Charles-François People requiring write durability will probably manage to find this full_sync parameter and if they do they thus really strive for data integrity, so call fsync() as a fallback for the security which Apple provides. Also: we cannot let os.fsync() fail with ENOTTY!? By the way, it's appropriate, not approbiate. You made the same typo in your patch. 8~I That was not a typo. Thanks. I'll changed that. -- Added file: http://bugs.python.org/file21986/11877.8.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -798,7 +798,7 @@ Availability: Unix. -.. function:: fsync(fd) +.. function:: fsync(fd, full_fsync=False) Force write of file with filedescriptor *fd* to disk. On Unix, this calls the native :c:func:`fsync` function; on Windows, the MS :c:func:`_commit` function. @@ -807,6 +807,15 @@ ``f.flush()``, and then do ``os.fsync(f.fileno())``, to ensure that all internal buffers associated with *f* are written to disk. + The POSIX standart requires that :c:func:`fsync` must transfer the buffered + data to the storage device, not that the data is actually written by the + device itself. It explicitely leaves it up to operating system implementors + wether users are given stronger guarantees on data integrity or not. Some + systems also offer special functions which overtake the part of making such + stronger guarantees, i.e., Mac OS X and NetBSD. The optional *full_fsync* + argument can be used to enforce usage of these special functions if that is + appropriate for the *fd* in question. + Availability: Unix, and Windows. diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -2121,13 +2121,50 @@ #ifdef HAVE_FSYNC PyDoc_STRVAR(posix_fsync__doc__, -fsync(fildes)\n\n\ -force write of file with filedescriptor to disk.); - -static PyObject * -posix_fsync(PyObject *self, PyObject *fdobj) -{ -return posix_fildes(fdobj, fsync); +fsync(fildes, full_fsync=False)\n\n +force write of file buffers with fildes to disk;\n +full_fsync forces flush of disk caches in case fsync() alone is not enough.); + +static PyObject * +posix_fsync(PyObject *self, PyObject *args, PyObject *kwargs) +{ +PyObject *fdobj; +int full_fsync = 0; +static char *keywords[] = {fd, full_fsync, NULL }; + +if (!PyArg_ParseTupleAndKeywords(args, kwargs, O|i, keywords, + fdobj, full_fsync)) +return NULL; + +/* See issue 11877 discussion */ +# if ((defined __APPLE__ defined F_FULLFSYNC) || \ + (defined __NetBSD__ defined FDISKSYNC)) +if (full_fsync != 0) { +int res, fd = PyObject_AsFileDescriptor(fdobj); +if (fd 0) +return NULL; +if (!_PyVerify_fd(fd)) +return posix_error(); + +Py_BEGIN_ALLOW_THREADS +# if defined __APPLE__ +/* F_FULLFSYNC is not supported for all types of FDs/FSYSs; + * be on the safe side and test for inappropriate ioctl errors */ +res = fcntl(fd, F_FULLFSYNC); +if (res 0 errno == ENOTTY) +res = fsync(fd); +# elif defined __NetBSD__ +res = fsync_range(fd, FFILESYNC|FDISKSYNC, 0, 0); +# endif +Py_END_ALLOW_THREADS + +if (res 0) +return posix_error(); +Py_INCREF(Py_None); +return Py_None; +} else +# endif +return posix_fildes(fdobj, fsync); } #endif /* HAVE_FSYNC */ @@ -9472,7 +9509,8 @@ {fchdir
[issue11877] Change os.fsync() to support physical backing store syncs
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file21973/11877.7.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: @Nir Aides: *thanks* for this link: http://groups.google.com/group/comp.programming.threads/msg/3a43122820983fde You made my day! -- nosy: +sdaoden ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11935] MMDF/MBOX mailbox need utime
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: For the record: On Mac OS X 10.6.7, ,HFS, case sensitive` updates st_atime by itself *once only*. It does so ~0.75 seconds after os.utime() (+) was called. A time.sleep(0.8) can be used to detect this automatic update reliably (about 50 tests with changing load all succeeded). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11935 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Ouch, ouch, ouch!! I'll have to send 11877.7.diff which extends 11877.6.diff. This is necessary because using fcntl(2) with F_FULLFSYNC may fail with ENOTTY (inapprobiate ioctl for device) in situations where a normal fsync(2) succeeds (e.g. STDOUT_FILENO). By the way - i have no idea of Redmoondian Horror at all (except for http://msdn.microsoft.com/en-us/sync/bb887623.aspx). Dropping .5 and .6 - and sorry for the noise. Good night, Europe. -- Added file: http://bugs.python.org/file21973/11877.7.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -798,7 +798,7 @@ Availability: Unix. -.. function:: fsync(fd) +.. function:: fsync(fd, full_fsync=False) Force write of file with filedescriptor *fd* to disk. On Unix, this calls the native :c:func:`fsync` function; on Windows, the MS :c:func:`_commit` function. @@ -807,6 +807,15 @@ ``f.flush()``, and then do ``os.fsync(f.fileno())``, to ensure that all internal buffers associated with *f* are written to disk. + The POSIX standart requires that :c:func:`fsync` must transfer the buffered + data to the storage device, not that the data is actually written by the + device itself. It explicitely leaves it up to operating system implementors + wether users are given stronger guarantees on data integrity or not. Some + systems also offer special functions which overtake the part of making such + stronger guarantees, i.e., Mac OS X and NetBSD. The optional *full_fsync* + argument can be used to enforce usage of these special functions if that is + approbiate for the *fd* in question. + Availability: Unix, and Windows. diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -2121,13 +2121,50 @@ #ifdef HAVE_FSYNC PyDoc_STRVAR(posix_fsync__doc__, -fsync(fildes)\n\n\ -force write of file with filedescriptor to disk.); - -static PyObject * -posix_fsync(PyObject *self, PyObject *fdobj) -{ -return posix_fildes(fdobj, fsync); +fsync(fildes, full_fsync=False)\n\n +force write of file buffers with fildes to disk;\n +full_fsync forces flush of disk caches in case fsync() alone is not enough.); + +static PyObject * +posix_fsync(PyObject *self, PyObject *args, PyObject *kwargs) +{ +PyObject *fdobj; +int full_fsync = 0; +static char *keywords[] = {fd, full_fsync, NULL }; + +if (!PyArg_ParseTupleAndKeywords(args, kwargs, O|i, keywords, + fdobj, full_fsync)) +return NULL; + +/* See issue 11877 discussion */ +# if ((defined __APPLE__ defined F_FULLFSYNC) || \ + (defined __NetBSD__ defined FDISKSYNC)) +if (full_fsync != 0) { +int res, fd = PyObject_AsFileDescriptor(fdobj); +if (fd 0) +return NULL; +if (!_PyVerify_fd(fd)) +return posix_error(); + +Py_BEGIN_ALLOW_THREADS +# if defined __APPLE__ +/* F_FULLFSYNC is not supported for all types of descriptors, be on the + * safe side and test for inapprobiate ioctl errors */ +res = fcntl(fd, F_FULLFSYNC); +if (res 0 errno == ENOTTY) +res = fsync(fd); +# elif defined __NetBSD__ +res = fsync_range(fd, FFILESYNC|FDISKSYNC, 0, 0); +# endif +Py_END_ALLOW_THREADS + +if (res 0) +return posix_error(); +Py_INCREF(Py_None); +return Py_None; +} else +# endif +return posix_fildes(fdobj, fsync); } #endif /* HAVE_FSYNC */ @@ -9472,7 +9509,8 @@ {fchdir, posix_fchdir, METH_O, posix_fchdir__doc__}, #endif #ifdef HAVE_FSYNC -{fsync, posix_fsync, METH_O, posix_fsync__doc__}, +{fsync, (PyCFunction)posix_fsync, METH_VARARGS|METH_KEYWORDS, +posix_fsync__doc__}, #endif #ifdef HAVE_SYNC {sync,posix_sync, METH_NOARGS, posix_sync__doc__}, ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file21924/11877.5.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file21953/11877.6.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: I don't agree with you and i don't believe it is implemented like that. But it seems i am the only one on this issue who sees it like that. Thus i apply 11877.6.diff. Declaring variables as auto is not necessary in C code and not used anywhere else in Python's source code Changed. Steffen, you changed the default to doing a full sync in your last patch Changed all through. The reason being that Apple and NetBSD folks should know what they're doing [.] People requiring write durability will probably manage to find this full_sync parameter. This sounds logical. I've changed the doc in os.rst so that it includes a note on *which* operating systems actually do something dependend on that argument. I should have done that from the very beginning. Finally, depending on the workload, it could have a significant performance impact. :-) -- Added file: http://bugs.python.org/file21953/11877.6.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -798,7 +798,7 @@ Availability: Unix. -.. function:: fsync(fd) +.. function:: fsync(fd, full_fsync=False) Force write of file with filedescriptor *fd* to disk. On Unix, this calls the native :c:func:`fsync` function; on Windows, the MS :c:func:`_commit` function. @@ -807,6 +807,13 @@ ``f.flush()``, and then do ``os.fsync(f.fileno())``, to ensure that all internal buffers associated with *f* are written to disk. + The POSIX standart only requires that :c:func:`fsync` must transfer the + buffered data to the storage device, not that the data is actually + written by the device itself. On operating systems where it is + necessary and possible the optional *full_fsync* argument can be used to + initiate additional steps to synchronize the physical backing store. + At the time of this writing this affects Apple Mac OS X and NetBSD. + Availability: Unix, and Windows. diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -2121,13 +2121,46 @@ #ifdef HAVE_FSYNC PyDoc_STRVAR(posix_fsync__doc__, -fsync(fildes)\n\n\ -force write of file with filedescriptor to disk.); - -static PyObject * -posix_fsync(PyObject *self, PyObject *fdobj) -{ -return posix_fildes(fdobj, fsync); +fsync(fildes, full_fsync=False)\n\n +force write of file buffers with fildes to disk;\n +full_fsync forces flush of disk caches in case fsync() alone is not enough.); + +static PyObject * +posix_fsync(PyObject *self, PyObject *args, PyObject *kwargs) +{ +PyObject *fdobj; +int full_fsync = 0; +static char *keywords[] = {fd, full_fsync, NULL }; + +if (!PyArg_ParseTupleAndKeywords(args, kwargs, O|i, keywords, + fdobj, full_fsync)) +return NULL; + +/* See issue 11877 discussion */ +# if ((defined __APPLE__ defined F_FULLFSYNC) || \ + (defined __NetBSD__ defined FDISKSYNC)) +if (full_fsync != 0) { +int res, fd = PyObject_AsFileDescriptor(fdobj); +if (fd 0) +return NULL; +if (!_PyVerify_fd(fd)) +return posix_error(); + +Py_BEGIN_ALLOW_THREADS +# if defined __APPLE__ +res = fcntl(fd, F_FULLFSYNC); +# elif defined __NetBSD__ +res = fsync_range(fd, FFILESYNC|FDISKSYNC, 0, 0); +# endif +Py_END_ALLOW_THREADS + +if (res 0) +return posix_error(); +Py_INCREF(Py_None); +return Py_None; +} else +# endif +return posix_fildes(fdobj, fsync); } #endif /* HAVE_FSYNC */ @@ -9472,7 +9505,8 @@ {fchdir, posix_fchdir, METH_O, posix_fchdir__doc__}, #endif #ifdef HAVE_FSYNC -{fsync, posix_fsync, METH_O, posix_fsync__doc__}, +{fsync, (PyCFunction)posix_fsync, METH_VARARGS|METH_KEYWORDS, +posix_fsync__doc__}, #endif #ifdef HAVE_SYNC {sync,posix_sync, METH_NOARGS, posix_sync__doc__}, ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Ronald Oussoren wrote (2011-05-08 10:33+0200): Steffen, I don't understand your comment about auto. Declaring variables as auto is not necessary in C code and not used anywhere else in Python's source code. Well, as long as i can keep my underwear all is fine. (I also looked at Google translate because i first wanted to start the reply with croak .. pip .. twist .. wrench .. groan .. ugh.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: @Nadeem: note that the committed versions of the tests would not show up the Mac OS X mmap() bug AFAIK, because there is an intermediate .close() of the file to be mmapped. The OS X bug is that the VMS/VFS interaction fails to provide a valid memory region for pages which are not yet physically present on disc - i.e. there is no true sparse file support as on Linux, which simply uses references to a single COW zero page. (I've not tried it out for real yet, but i'm foolish like a prowd cock, so i've looked at the changeset :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: (Of course this may also be intentional, say. But then i would vote against it :), because it's better the tests bring out errors than end-user apps.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11999] sporadic failure in test_mailbox on FreeBSD
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: That's really a good one. (In my eyes.) -- title: sporadic failure in test_mailbox - sporadic failure in test_mailbox on FreeBSD ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11999 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: On Sat, 7 May 2011 14:20:51 +0200, Charles-François Natali wrote: I really can't see a good reason for using it here (and anywhere, see http://c-faq.com/decl/auto.html). You're right. Why are you using the auto storage class specifier? I know that explicit is better than implicit Yup. I'm doing what is happening for real in (x86) assembler. Thus auto means (at a glance!) that this one will need space on the stack. (Conversely i'm not using register because who knows if that is really true? ;)) Do you really need to use goto for such a simple code? Yup. Ok, this is more complicated. The reason is that my funs have exactly one entry point and exactly one place where they're left. This is because we here do manual instrumentalisation as in ret fun(args) { locals s_NYD_ENTER; assertions ... jleave: s_NYD_LEAVE; return; [maybe only, if large: possibly predict-false code blocks] [possible error jumps here goto jleave;] } We're prowd of that. N(ot)Y(et)D(ead) is actually pretty cool, i've used debuggers exactly 5 times in the past about ten years! I don't even know exactly *how to use debuggers*. 8---} (NYD can do backtracing, or, with NDEBUG and optional, profiling.) A really good optimizing compiler will produce the very same code! And i love nasm, but it's pretty non-portable. But C is also nice. But of course i can change this (in C) to simply use return, this is easy here, no resources to be freed. Thanks for looking at this, by the way. :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: On Sat, 7 May 2011 14:20:51 +0200, Charles-François Natali wrote: # ifdef __APPLE__ res = fcntl(fd, F_FULLFSYNC); # endif # ifdef __NetBSD__ res = fsync_range(fd, FFILESYNC|FDISKSYNC, 0, 0); # endif Since __APPLE__ and __NetBSD__ are exclusive, you could use something like # if defined(__APPLE__) res = fcntl(fd, F_FULLFSYNC); # elif defined(__NetBSD__) res = fsync_range(fd, FFILESYNC|FDISKSYNC, 0, 0); # endif Yes, you're right, i'll update the patch accordingly. -- Steffen, sdaoden(*)(gmail.com) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11999] sporadic failure in test_mailbox
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: This tracker - sorry! (I surely will try to write and offer a patch for the tracker, but first i need to understand that mailbox.py jungle for #11935, next week.) -- title: sporadic failure in test_mailbox on FreeBSD - sporadic failure in test_mailbox ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11999 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: 11877.5.diff incorporates all changes suggested by Charles-François except for the 'auto' keyword, which is extremely important and which would need to be invented if it would not yet exist. I'm dropping the old stuff. And i think this is the final version of the patch. I've changed the default argument to 'True' as i really think it's better to be on the safe side here. (The french are better off doing some gracy and dangerous sports to discover edges of life!) I'm still of the opinion that this should be completely hidden, but since it's completely transparent wether a Python function gets yet another named argument or not... So, thanks to Ronald, i detected that also NetBSD introduced a FDISKSYNC flag in 2005 and that you really need fsync_range() there (at least by definition)! How could they do that? But i'm also happy to see that all other systems try hard to achieve security transparently and by default and unless i missed something once again. -- Added file: http://bugs.python.org/file21924/11877.5.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -798,7 +798,7 @@ Availability: Unix. -.. function:: fsync(fd) +.. function:: fsync(fd, full_fsync=True) Force write of file with filedescriptor *fd* to disk. On Unix, this calls the native :c:func:`fsync` function; on Windows, the MS :c:func:`_commit` function. @@ -807,6 +807,12 @@ ``f.flush()``, and then do ``os.fsync(f.fileno())``, to ensure that all internal buffers associated with *f* are written to disk. + The POSIX standart only requires that :c:func:`fsync` must transfer the + buffered data to the storage device, not that the data is actually + written by the device itself. On operating systems where it is + necessary and possible the optional *full_fsync* argument can be used to + initiate additional steps to synchronize the physical backing store. + Availability: Unix, and Windows. diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -2121,13 +2121,46 @@ #ifdef HAVE_FSYNC PyDoc_STRVAR(posix_fsync__doc__, -fsync(fildes)\n\n\ -force write of file with filedescriptor to disk.); - -static PyObject * -posix_fsync(PyObject *self, PyObject *fdobj) -{ -return posix_fildes(fdobj, fsync); +fsync(fildes, full_fsync=True)\n\n +force write of file buffers with fildes to disk;\n +full_fsync forces flush of disk caches in case fsync() alone is not enough.); + +static PyObject * +posix_fsync(PyObject *self, PyObject *args, PyObject *kwargs) +{ +auto PyObject *fdobj; +auto int full_fsync = 1; +static char *keywords[] = {fd, full_fsync, NULL }; + +if (!PyArg_ParseTupleAndKeywords(args, kwargs, O|i, keywords, + fdobj, full_fsync)) +return NULL; + +/* See issue 11877 discussion */ +# if ((defined __APPLE__ defined F_FULLFSYNC) || \ + (defined __NetBSD__ defined FDISKSYNC)) +if (full_fsync != 0) { +int res, fd = PyObject_AsFileDescriptor(fdobj); +if (fd 0) +return NULL; +if (!_PyVerify_fd(fd)) +return posix_error(); + +Py_BEGIN_ALLOW_THREADS +# if defined __APPLE__ +res = fcntl(fd, F_FULLFSYNC); +# elif defined __NetBSD__ +res = fsync_range(fd, FFILESYNC|FDISKSYNC, 0, 0); +# endif +Py_END_ALLOW_THREADS + +if (res 0) +return posix_error(); +Py_INCREF(Py_None); +return Py_None; +} else +# endif +return posix_fildes(fdobj, fsync); } #endif /* HAVE_FSYNC */ @@ -9472,7 +9505,8 @@ {fchdir, posix_fchdir, METH_O, posix_fchdir__doc__}, #endif #ifdef HAVE_FSYNC -{fsync, posix_fsync, METH_O, posix_fsync__doc__}, +{fsync, (PyCFunction)posix_fsync, METH_VARARGS|METH_KEYWORDS, +posix_fsync__doc__}, #endif #ifdef HAVE_SYNC {sync,posix_sync, METH_NOARGS, posix_sync__doc__}, ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file21749/11877.3.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file21771/11877.4.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11877 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11999] sporadic failure in test_mailbox on FreeBSD
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: On Fri, 6 May 2011 04:44:00 +0200, R. David Murray wrote: [.] the mtime only has a resolution of one second. You always say that! But i'm pretty sure from somewhen long ago that there are filesystems which have a two second time resolution. And unless i'm mistaken that filesystem is still used widely. Attached is a patch implementing the fix. It undoes the 6896 patch I've not yet tried your code but from looking at the patch it seems to target towards a real approach. I also added an additional delta in case the file system clock is skewing relative to the system clock. I made this a class attribute so that it is adjustable; perhaps it should be made public and documented. On the other hand, if it shows up after almost five years that the one second resolution solution doesn't work, and that simply adjusting to a two second resolution is not smart enough to get this fixed, then i would not go for something half-automatic which a user needs to adjust manually, because how could a user do that? Thus, in my view, if you are *really* looking forward to make mailbox.py a *good* and *beautiful* thing then the timedelta in between the filesystem and the host's time() must of course be tracked-, and the fuzziness should be adjusted automatically. E.g. similar http://linuxcommand.org/man_pages/adjtimex8.html. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11999 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11999] sporadic failure in test_mailbox on FreeBSD
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: I also added an additional delta in case the file system clock is skewing relative to the system clock. In fact this idea could even be made public e.g. like this class ClockDrifter: def add_snapshot(self, exttime, loctime=None): if loctime is None: loctime = time.time() ... def drift_tendency(self): ... def drift_distance(self): ... I could think of usages thereof. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11999 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11999] sporadic failure in test_mailbox on FreeBSD
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: I like all kind of single-screw solutions. And i feel a bit uncomfortable with your usual tightness. (0.1 seconds - i mean, come on) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11999 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11999] sporadic failure in test_mailbox on FreeBSD
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: how many people are going to use maildir on FAT? Dunno. But it's maybe the lowest common denominator of mountable readwrite filesystems. Except for that MacBook i always had a shared FAT partition on my private PCs. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11999 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11935] MMDF/MBOX mailbox need utime
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: @david: note i got stuck on updating my patch for mailbox.py and switched to do test_mmap.py instead, so that i don't know wether i will be able to finish it today. Is it really true that mailbox.py even writes mailboxes without locking in case of an appending write? So i really have to look at that before i will proceed and write the patch. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11935 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11999] sporadic failure in test_mailbox on FreeBSD
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: I agree that it's fun to see pieces of code interacting like gears in a transmission, but often it gets ugly due to the noise from the outside which requires to add ugly fuzziness or honour stupid design decisions. Maybe an environment variable like PYMAILBOX_SKEW and then one could run test_mailbox.py and see if it works for 0.01?? :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11999 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: On Fri, 6 May 2011 02:54:07 +0200, Nadeem Vawda wrote: I think so. [.] it turns out that the OS X sparsefile crash is also covered by LargeMmapTests.test_large_offset() in test_mmap [!!!]. [.] So i followed your suggestion and did not do something on zlib no more. Even if that means that there is no test which checksums an entire superlarge mmap() region. Instead i've changed/added test cases in test_mmap.py: - Removed all context-manager usage from LargeMmapTests(). This feature has been introduced in 3.2 and is already tested elsewhere. Like this the test is almost identical on 2.7 and 3.x. - I've dropped _working_largefile(). This creates a useless large file only to unlink it directly. Instead the necessary try:catch: is done directly in the tests. - (Directly testing after .flush() without reopening the file.) - These new tests don't run on 32 bit. May the juice be with you -- Added file: http://bugs.python.org/file21909/11277-test_mmap.1.py Added file: http://bugs.python.org/file21910/11277-test_mmap-27.1.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___diff --git a/Lib/test/test_mmap.py b/Lib/test/test_mmap.py --- a/Lib/test/test_mmap.py +++ b/Lib/test/test_mmap.py @@ -1,4 +1,5 @@ -from test.support import TESTFN, run_unittest, import_module, unlink, requires +from test.support import TESTFN, run_unittest, import_module, unlink +from test.support import requires, _4G import unittest import os import re @@ -662,44 +663,87 @@ def tearDown(self): unlink(TESTFN) -def _working_largefile(self): -# Only run if the current filesystem supports large files. -f = open(TESTFN, 'wb', buffering=0) +def _test_splice(self, f, i): +# Test splicing with pages around critical values in respect to +# memory management +# Issue 11277: does mmap() force materialization of backing store? +m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) try: -f.seek(0x8001) -f.write(b'x') -f.flush() -except (IOError, OverflowError): -raise unittest.SkipTest(filesystem does not have largefile support) +# Memory page before xy +self.assertEqual(m[i+0:i+2], b' ') +# Memory page after xy +self.assertEqual(m[i+10:i+12], b' ') +# Cross pages +self.assertEqual(m[i+2:i+10], b'DEARdear') finally: -f.close() -unlink(TESTFN) +m.close() -def test_large_offset(self): +def _test_subscr(self, f, idx, expect): +# Test subscript for critical values like INT32_MAX, UINT32_MAX +m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) +try: +self.assertEqual(m[idx], expect) +finally: +m.close() + +@unittest.skipUnless(sys.maxsize _4G, Can't run on a 32-bit system.) +def test_around_32bit_sbitflip(self): +start = 0x7FFA if sys.platform[:3] == 'win' or sys.platform == 'darwin': requires('largefile', -'test requires %s bytes and a long time to run' % str(0x18000)) -self._working_largefile() -with open(TESTFN, 'wb') as f: -f.seek(0x14FFF) -f.write(b ) + 'test requires %s bytes and a long time to run' % + str(start+12)) +with open(TESTFN, 'w+b') as f: +try: +f.seek(start) +f.write(b' DEARdear ') +f.flush() +except (IOError, OverflowError): +raise unittest.SkipTest('filesystem does not have largefile ' +'support') +self._test_splice(f, start) +self._test_subscr(f, start+len(b' DEA'), ord(b'R')) +self._test_subscr(f, start+len(b' DEARdea'), ord(b'r')) +unlink(TESTFN) -with open(TESTFN, 'rb') as f: -with mmap.mmap(f.fileno(), 0, offset=0x14000, access=mmap.ACCESS_READ) as m: -self.assertEqual(m[0xFFF], 32) +@unittest.skipUnless(sys.maxsize _4G, Can't run on a 32-bit system.) +def test_around_32bit_excess(self): +start = 0xFFFA +if sys.platform[:3] == 'win' or sys.platform == 'darwin': +requires('largefile', + 'test requires %s bytes and a long time to run' % + str(start+12)) +with open(TESTFN, 'w+b') as f: +try: +f.seek(start) +f.write(b' DEARdear ') +f.flush() +except (IOError, OverflowError): +raise unittest.SkipTest('filesystem does not have largefile
[issue11277] Crash with mmap and sparse files on Mac OS X
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file21869/11277-27.2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file21885/11277-27.3.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11935] MMDF/MBOX mailbox need utime
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: On Thu, 5 May 2011 03:52:29 +0200, R. David Murray wrote: [..] the shell [..] I believe it just looks at the mtime/atime. /* check_mail () is useful for more than just checking mail. Since it has the paranoids dream ability of telling you when someone has read your mail, it can just as easily be used to tell you when someones .profile file has been read, thus letting one know when someone else has logged in. Pretty good, huh? */ /* If the user has just run a program which manipulates the mail file, then don't bother explaining that the mail file has been manipulated. Since some systems don't change the access time to be equal to the modification time when the mail in the file is manipulated, check the size also. If the file has not grown, continue. */ /* If the mod time is later than the access time and the file has grown, note the fact that this is *new* mail. */ Not all system mail spools are mode 1777. Mutt needs to be setgid mail on systems that aren't, if I understand correctly. Making a python program setgid mail is a bit more of security issue than making a well-tested C program setgid, since it is easier to break out of the box in a python program. Ok, maybe set-group-ID on /var/mail isn't even necessary; 0 drwxrwxx-x3 root mail 102 5 May 11:30 mail is enough as long as $ groups $USER states you are member of group mail. On my system mailbox.py doesn't have any problems with modifying the mail directory. If this is not true on your box go and stress your admin, he's not worth his money - is he? I.e., whereas it is possible to rewrite mailbox.py to handle issue #7359 i would not do so because it is unnecessary on correctly setup boxes. Maybe mailbox.py has used so much copy-and-paste from mutt(1)'s mbox.c because that code works well for many years. And Jason seems to work as root all of the time. mailbox is an mbox manipulation program, not a mail delivery agent. If you are using it to write a mail delivery agent, I think perhaps the mtime setting code belongs in your application, not the mailbox module. I really don't understand your point now. Of course the standart is soft like butter in that it seems to assume that the spool mailbox is then locally processed and truncated to zero length, so that mailbox has grown==new mail arrived, whereas it is also possible to use that spool file as a real local mailbox, including resorting, partial deletion etc.. This issue is about fixing mailbox.py to adhere MMDF and MBOX standarts, which is what the patch implements. This patch works for me locally in that mutt(1) will mention that new mail has arrived in the boxes. The patch uses a safe approach by dating back the access time instead of pointing modification time into the future, which however will make the patch fail on Pear OS X if the mailbox is on HFS, case sensitive, because that is buggy and *always* updates atime; maybe this is because Apple only provides a shallow wrapper around UFS to integrate that in the Microkernel/IOKit structure, just in case HFS, case sensitive is really UFS, but i'm guessing in all directions here. I would not adjust the patch to fix this, but the problem exists and it has been noted in this issue. -- Steffen sdao...@gmail.com -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11935 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11935] MMDF/MBOX mailbox need utime
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: On Thu, 5 May 2011 03:52:29 +0200, R. David Murray wrote: [..] the shell [..] I believe it just looks at the mtime/atime. Pretty good, huh? Mr. Mojo says: Prowd to be a part of this number. Successful hills are here to stay. Everything must be this way. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11935 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: @haypo: trouble, trouble on the dev-list, as i've seen currently. Sorry, sorry. (Cannot subscribe, my DynIP's are often blacklisted ;) Of course my comments were completely wrong, as Ethan has pointed out correctly. This is all s**t. These are mmap(2) related issues and should be tested in Lib/test/test_mmap.py. However that does not use with open: create sparse file materialize yet so that the Pear OS X sparsefile bug doesn't show up. In fact it doesn't do a full beam-me-up test at all yet? Is the test useful or not? What do we test? We do test that mmap.mmap materializes a buffer which can be accessed (readonly) from [0]..[len-1]. And that the checksums that zlib produces for that buffer are correct. Unfortunately we cannot test 0x8000+ no more because Python prevents that such a buffer can be used - that's a shame. Maybe we could test 0x7FFF*2 aka 0xfffe in two iterations. Can you check if the test crashs on Mac OS X on a 32 bits system (1 GB buffer) if you disable F_FULLFSYNC in mmapmodule.c? Same question on a 64 bits system (2 GB-1 byte buffer)? Aeh - F_FULLFSYNC was not yet committed at that time in 2.7. Can we just remove the test? If i - choke! - need to write tests, i try to catch corner cases. The corner cases would be 0,MAX_LEN(-1) and some (rather pseudo) random values around these and maybe some in the middle. (Plus some invalid inputs.) Can we remove it? I would keep it, Apple is preparing the next major release (uname -a yet states 10.7.0 even though it's 10.6.7), and maybe then mmap() will fail for 0xDEADBEEF. Who will be the one which detects that otherwise?? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12000] SSL certificate verification failed if no dNSName entry in subjectAltName
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: P.S.: if you're really right ('have those RFC's, but didn't read them yet), you could also open an issue for Mercurial at http://mercurial.selenic.com/bts - i think those guys do the very same. Thanks, Steffen! -- nosy: +sdaoden ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12000 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11935] MMDF/MBOX mailbox need utime
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: On Thu, 5 May 2011 13:42:29 +0200, R. David Murray wrote: what does mutt do in the case you are talking about? 16 -rwxr-s--- 1 steffen mail 14832 23 Jan 19:13 usr/bin/mutt_bitlock set bitlock_program=~/usr/bin/mutt_bitlock -p I see. Unfortunately the world is not even almost perfect. So should f?truncate(2) be used if the resulting file is empty? what does mutt do in the case you are talking about? Otherwise there is only one solution: a mailbox-is-readonly policy has to be introduced. That will surely drive users insane which see that they in fact have write access to the file. Python has got bad cards. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11935 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11935] MMDF/MBOX mailbox need utime
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: The problem report in question was submitted by one of the Debian maintainers. Yeah, a documentainer at least. I've used Debian (Woody i think that was 3.1). Actually great because Lehmanns, Heidelberg, Germany did not include the sources but they've sent me the sources (on seven CD's as far as i remember) for free after i've complained. Linux is really great. You don't need internet access at all because of that fantastic documentation, everywhere. You look into /dev and /sys and /proc and it's all so translucent!! And the GNU tools and libraries - they are so nicely designed. The source code is so clean. It's really an enlightened system. Then i discovered FreeBSD 4.8 which released me from all that. \|/ _ . | - | (I still had hairs at that time. But that was long ago.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11935 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11935] MMDF/MBOX mailbox need utime
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: After half an hour of shallow inspection. mutt really modifies mailbox files in place (mbox_sync_mailbox()) after creating (all the new tail in) a temporary file. Then seek()/write()/truncate() etc.. It however has mutt_dotlock(1) and it does block signals and it is a standalone program and thus i don't think this behaviour can be used by Python. In respect to our issue here i must really admit that mutt does: prepare new tail stat box modify box to incorporate tail close box utime box with stat result times reopen box So actually the result looks as if it never has been modified. But maybe it is because like this it is in sync with the standart, since strictly speaking there is no *new* mail in the box. Unless you vote against it i'll write a patch tomorrow which will use a state machine which only triggers the utime if some kind of setitem has occurred. I can't help you to overcome your malaise against soiling an atime's pureness. 'Really want a future date?? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11935 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: In fact i like my idea of using iterations. I have some time tomorrow, so if nobody complains until then, i write diffs for the tests of 3.x and 2.7 with these updates: - Two different target sizes: 1. 0x + x (7) 2. 0x7FFF + x (7) - On 32 bit systems, use iterations on a potentially safe buffer size. I think 0x4000 a.k.a 1024*1024*1024 is affordable, but 512 MB are probably more safe? I'll make that a variable. - The string will be 'DeadAffe' (8). - The last 4 bytes of the string will always be read on their own (just in case the large buffer sizes irritated something down the path). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7359] mailbox cannot modify mailboxes in system mail spool
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Issue #11935 becomes gigantic and may even swamp this one over! -- nosy: +sdaoden ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7359 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11935] MMDF/MBOX mailbox need utime
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: On Thu, 5 May 2011 19:04:16 +0200, R. David Murray wrote: prepare new tail means all of the text from the first modified line to the end? (As opposed to just the new mail?) mailbox does locking. I see no reason in principle it couldn't stat/restore, it would just be setting the times on the new file rather than on a truncated/rewritten old file. How hard that would be to incorporate into the existing logic I have no idea. Of course there may be issues about this I haven't thought of. Me too and even more. Clearly mailbox.py cannot do any dotlocking due to missing permissions, so this is silently ignored to be able to proceed at all. Therefore only fcntl/flock locking is used for a /var/{spool/}mail box by mailbox.py. This is fine as long as all programs agree in locking such a file in the usual way, that is, use both, dotlocking *and* flock/lock, and restart from the beginning if one of them fails due to another program holding that very lock. mutt does that but i won't do any bet here. And then the signal handling, and Python even supports threading, and it is embeddable and there may be third-party modules also involved. This is the Death Valley of programming. $PYP mb.flush() Traceback (most recent call last): File stdin, line 1, in module File /Users/steffen/usr/opt/py3k/lib/python3.3/mailbox.py, line 659, in flush new_file = _create_temporary(self._path) File /Users/steffen/usr/opt/py3k/lib/python3.3/mailbox.py, line 2061, in _create_temporary os.getpid())) File /Users/steffen/usr/opt/py3k/lib/python3.3/mailbox.py, line 2051, in _create_carefully fd = os.open(path, os.O_CREAT | os.O_EXCL | os.O_RDWR, 0o666) OSError: [Errno 13] Permission denied: '/var/mail/steffen.1304628960.sherwood.local.37135' So this seems to be the safest and most useful approach in this context, because i do not want to imagine what happens if something weird occurs in the middle of writing the tail otherwise. So i stop thinking about issue #7359. Do you think the mutt model is a good one to follow? You mean resetting atime/mtime back to before the rename? I don't like that and i don't understand it because the file has been modified, so i think i would do (now,now) in that case instead (because of the MMDF/MBOX newer==new mail case). And in case a new mail has been inserted (now-2.42,now). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11935 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11997] One typo in Doc/c-api/init.rst
New submission from Steffen Daode Nurpmeso sdao...@googlemail.com: Yes, i really found a typo. I'll send two patches, one with the typo fixed, and one with the typo fixed and one for which i've :setlocal tw=80:{gq} -- assignee: docs@python components: Documentation messages: 135107 nosy: docs@python, sdaoden priority: normal severity: normal status: open title: One typo in Doc/c-api/init.rst versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11997 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11997] One typo in Doc/c-api/init.rst
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: 11997.1.diff only corrects the typo, 11997.2.diff does also reformat. (Note that all of init.rst is hard to read on a 80 column terminal.) -- keywords: +patch Added file: http://bugs.python.org/file21880/11997.1.diff Added file: http://bugs.python.org/file21881/11997.2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11997 ___diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -883,7 +883,7 @@ modules. Also note that combining this functionality with :c:func:`PyGILState_\*` APIs -is delicate, become these APIs assume a bijection between Python thread states +is delicate, because these APIs assume a bijection between Python thread states and OS-level threads, an assumption broken by the presence of sub-interpreters. It is highly recommended that you don't switch sub-interpreters between a pair of matching :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release` calls. diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -882,14 +882,14 @@ by such objects may affect the wrong (sub-)interpreter's dictionary of loaded modules. -Also note that combining this functionality with :c:func:`PyGILState_\*` APIs -is delicate, become these APIs assume a bijection between Python thread states -and OS-level threads, an assumption broken by the presence of sub-interpreters. -It is highly recommended that you don't switch sub-interpreters between a pair -of matching :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release` calls. -Furthermore, extensions (such as :mod:`ctypes`) using these APIs to allow calling -of Python code from non-Python created threads will probably be broken when using -sub-interpreters. +Also note that combining this functionality with :c:func:`PyGILState_\*` APIs is +delicate, because these APIs assume a bijection between Python thread states and +OS-level threads, an assumption broken by the presence of sub-interpreters. It +is highly recommended that you don't switch sub-interpreters between a pair of +matching :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release` calls. +Furthermore, extensions (such as :mod:`ctypes`) using these APIs to allow +calling of Python code from non-Python created threads will probably be broken +when using sub-interpreters. Asynchronous Notifications ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11999] sporadic failure in test_mailbox on FreeBSD
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: I think this relates #6896. Maybe a two second resolution should be tried? -- keywords: +patch nosy: +sdaoden Added file: http://bugs.python.org/file21884/11999.1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11999 ___diff --git a/Lib/mailbox.py b/Lib/mailbox.py --- a/Lib/mailbox.py +++ b/Lib/mailbox.py @@ -514,13 +514,11 @@ else: return -# We record the current time - 1sec so that, if _refresh() is called -# again in the same second, we will always re-read the mailbox -# just in case it's been modified. (os.path.mtime() only has -# 1sec resolution.) This results in a few unnecessary re-reads -# when _refresh() is called multiple times in the same second, -# but once the clock ticks over, we will only re-read as needed. -now = time.time() - 1 +# Try to be fancy by using a date in the past for our _last_read mtime +# checks (see issues #6896, #11999) +# Using a two second resolution should be enough to overcome all +# fuzziness which may be introduced along the different filesystems. +now = time.time() - 2 self._toc = {} def update_dir (subdir): ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: error: [Errno 12] Cannot allocate memory @haypo: Well i told you i have no idea. These bots are 32 bit? I'll attach 11277-27.3.diff which does @skipUnless(not 32 bit). Note i'll test against _4G - does this work (on 32 bit and in Python)? A pity that Python does not offer a 'condition is always true due to datatype storage restriction' check?! And i don't think it makes sense to test a _1GB mmap on 32 bit at all (but at least address space shouldn't exhaust for that). So, sorry, also for the two bugs in that two-liner, but very especially the 'm' case. -- Added file: http://bugs.python.org/file21885/11277-27.3.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___diff --git a/Lib/test/test_zlib.py b/Lib/test/test_zlib.py --- a/Lib/test/test_zlib.py +++ b/Lib/test/test_zlib.py @@ -2,7 +2,7 @@ from test.test_support import TESTFN, run_unittest, import_module, unlink, requires import binascii import random -from test.test_support import precisionbigmemtest, _1G +from test.test_support import precisionbigmemtest, _1G, _2G, _4G import sys try: @@ -75,17 +75,16 @@ # Issue #11277 - check that inputs of 2 GB are handled correctly. # Be aware of issues #1202, #8650, #8651 and #10276 class ChecksumBigBufferTestCase(unittest.TestCase): -int_max = 0x7FFF - +@unittest.skipUnless(sys.maxsize _4G, Can't run on a 32-bit system.) @unittest.skipUnless(mmap, mmap() is not available.) def test_big_buffer(self): if sys.platform[:3] == 'win' or sys.platform == 'darwin': requires('largefile', 'test requires %s bytes and a long time to run' % - str(self.int_max)) + str(_2G -1)) try: with open(TESTFN, wb+) as f: -f.seek(self.int_max-4) +f.seek(_2G -1-4) f.write(asdf) f.flush() m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: @haypo: Oh. Not: if sys.maxsize _4G: # (64 bits system) crc32() and adler32() stores the buffer size into an # int, the maximum filesize is INT_MAX (0x7FFF) filesize = 0x7FFF crc_res = 0x709418e7 adler_res = -2072837729 else: # (32 bits system) On a 32 bits OS, a process cannot usually address # more than 2 GB, so test only 1 GB filesize = _1G crc_res = 0x2b09ee11 adler_res = -1002962529 self.assertEqual(zlib.crc32(m), self.crc_res) self.assertEqual(zlib.adler32(m), self.adler_res) I'm not that fast. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10044] small int optimization
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Now me. (http://gcc.gnu.org/onlinedocs/gcc/Arrays-and-pointers-implementation.html#Arrays-and-pointers-implementation) When casting from pointer to integer and back again, the resulting pointer must reference the same object as the original pointer, otherwise the behavior is undefined. That is, one may not use integer arithmetic to avoid the undefined behavior of pointer arithmetic as proscribed in C99 6.5.6/8. Say - isn't that a joke about farts of armchair crouchers. All it says is that if you dereference garbage you get a crash. If you're concerned, use volatile, and go shoot the compiler programmer if she dares to optimize just anything. And on ARM, isn't the interrupt table at ((void*)(char[])0x0)?? La vie est une rose beside that. And all i need is atomic_compare_and_swap(). -- nosy: +sdaoden ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10044 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Should we fix Python 2.7? - backport issue #8651 - use PY_SSIZE_T_CLEAN in zlibmodule.c I really thought about this over night. I'm a C programmer and thus: - Produce no bugs - If you've produced a bug, fix it at once - If you've fixed a bug, scream out loud BUGFIX! - or at least incorporate the patch in the very next patch release But i have no experience with maintaining a scripting language. My survey of something like this spans about three months now. And if even such a heavy known bug as #1202 survives at least two minor releases (2.6 and 2.7) without being fixed, then maybe no more effort should be put into 2.7 at all. 11277-27.1.diff contains # Issue #10276 - check that inputs =4GB are handled correctly.. I don't understand this comment because the test uses a buffer of 2 GB + 2 bytes. How is it possible to pass a buffer of 2 GB+2 bytes to crc32(), whereas it stores the size into an int. The maximum size is INT_MAX which is 2 GB-1 byte. It looks like the i format of PyArg_ParseTuple() doesn't check for integer overflow = issue #8651. This issue was fixed in 3.1, 3.2 and 3.3, but not in Python 2 11277-27.2.diff uses INT_MAX and thus avoids any such pitfall. Maybe it brings up memory mapping errors somewhere which i surely would try fix everywhere i can. -- Added file: http://bugs.python.org/file21869/11277-27.2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___diff --git a/Lib/test/test_zlib.py b/Lib/test/test_zlib.py --- a/Lib/test/test_zlib.py +++ b/Lib/test/test_zlib.py @@ -1,10 +1,16 @@ import unittest -from test import test_support +from test.test_support import TESTFN, run_unittest, import_module, unlink, requires import binascii import random from test.test_support import precisionbigmemtest, _1G +import sys -zlib = test_support.import_module('zlib') +try: +import mmap +except ImportError: +mmap = None + +zlib = import_module('zlib') class ChecksumTestCase(unittest.TestCase): @@ -66,6 +72,34 @@ zlib.crc32('spam', (2**31))) +# Backport to 2.7 due to Issue #11277: why not also verify INT32_MAX on 2.7? +# Be aware of issues #1202, #8650, #8651 and #10276 +class ChecksumBigBufferTestCase(unittest.TestCase): +int_max = 0x7FFF + +@unittest.skipUnless(mmap, mmap() is not available.) +def test_big_buffer(self): +if sys.platform[:3] == 'win' or sys.platform == 'darwin': +requires('largefile', + 'test requires %s bytes and a long time to run' % + str(self.int_max)) +try: +with open(TESTFN, wb+) as f: +f.seek(self.int_max-4) +f.write(asdf) +f.flush() +try: +m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) +self.assertEqual(zlib.crc32(m), 0x709418e7) +self.assertEqual(zlib.adler32(m), -2072837729) +finally: +m.close() +except (IOError, OverflowError): +raise unittest.SkipTest(filesystem doesn't have largefile support) +finally: +unlink(TESTFN) + + class ExceptionTestCase(unittest.TestCase): # make sure we generate some expected errors def test_badlevel(self): @@ -546,8 +580,9 @@ def test_main(): -test_support.run_unittest( +run_unittest( ChecksumTestCase, +ChecksumBigBufferTestCase, ExceptionTestCase, CompressTestCase, CompressObjectTestCase ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file21855/11277-27.1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10276] zlib crc32/adler32 buffer length truncation (64-bit)
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Ha! I always knew it! wink -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10276 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11935] MMDF/MBOX mailbox need utime
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: On Sun, 1 May 2011 00:15:11 +0200, R. David Murray rep...@bugs.python.org wrote: The problem with this patch is that it would also show 'new mail' if what had in fact happened was that a message had been *deleted* (see the comments at the beginning of the flush method). So actually fixing this is a bit more complicated. Well i don't think so because MUA's do some further checks, like checking the size and of course the status of each mail; some indeed use the mtime as an entry-gate for further inspection. And deleting an entry should surely pass that gate, too. Please do see the file mbox.c of the mutt(1) source repository, which in fact seems to have been used as an almost copy-and-paste template for the implementation of large parts of mailbox.py. But note that i just search less than five minutes in mailbox.py to find a place where i can add the code of the patch (after i've added an identical workaround for my S-Postman), so of course it may not catch all cases. One error is obvious: it also sets the mtime for that Babylon format. I don't use emacs and i'm a buddhist so i don't care about that Babylon mess anyway. Right? A proper fix for this should also consider fixing issue 7359. Hm. #7359 refers to misconfiguration, not to Python or mailbox.py. Maybe Doc/library/mailbox.rst should be adjusted to give users which are new to UNIX a hint about ,group mail` and the set-group-ID on directories? I think this would really be a good thing?!?! Should i open an issue on that? But again, mailbox.py reflects almost one-to-one (except for the naive file lock handling in comparison and AFAIK) mutt(1)'s mbox.c, and i think that if mutt(1) does create-temp-work-work-work-rename then this should be ok for mailbox.py, too. Did you know that ,bin Laden` ment ,am loading` in german? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11935 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11935] MMDF/MBOX mailbox need utime
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: I'll attach a patch with a clearer comment (entry-gate instead new mail), i.e. the comment now reflects what MUAs really do. -- Added file: http://bugs.python.org/file21850/11935.2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11935 ___diff --git a/Lib/mailbox.py b/Lib/mailbox.py --- a/Lib/mailbox.py +++ b/Lib/mailbox.py @@ -692,6 +692,13 @@ self._file = open(self._path, 'rb+') self._toc = new_toc self._pending = False +# Set modification time to be after access time so that MMDF and MBOX +# mail readers detect changes (or perform further inspection to do so) +try: +currtime = time.time() +os.utime(self._path, (currtime-3, currtime)) +except: +pass if self._locked: _lock_file(self._file, dotlock=False) ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11935] MMDF/MBOX mailbox need utime
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file21795/mailbox.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11935 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: On Mon, 2 May 2011 01:22:41 +0200, STINNER Victor rep...@bugs.python.org wrote: @sdaoden: Can you try on Python 2.7? @haypo: Python 2.7 is absolute horror. But i tried and produced a (terrible - i don't know the test framework and that test_support stuff seems to have been changed a lot since 2.7) 2 gigabyte+ big buffer test for 2.7. (Of course: even though Python uses int, ZLib uses uInt.) It took some time because i fell over #1202 from 2007 unprepared. The (nasty) test works quite well on Apple, which is not such a big surprise, because Apple's OS X is especially designed for artists which need to work on large files, like video+ cutters, sound designers with sample databases etc., so i would be terribly disappointed if that wouldn't work! Apple even propagandize OS X for, and makes money with that very application task - i really couldn't understand your doubts here. -- Added file: http://bugs.python.org/file21855/11277-27.1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___diff --git a/Lib/test/test_zlib.py b/Lib/test/test_zlib.py --- a/Lib/test/test_zlib.py +++ b/Lib/test/test_zlib.py @@ -1,10 +1,16 @@ import unittest -from test import test_support +from test.test_support import TESTFN, run_unittest, import_module, unlink, requires +from test.test_support import precisionbigmemtest, _1G, _2G import binascii import random -from test.test_support import precisionbigmemtest, _1G +import sys -zlib = test_support.import_module('zlib') +try: +import mmap +except ImportError: +mmap = None + +zlib = import_module('zlib') class ChecksumTestCase(unittest.TestCase): @@ -66,6 +72,32 @@ zlib.crc32('spam', (2**31))) +# Issue #10276 - check that inputs =4GB are handled correctly. +# Backport to 2.7 due to Issue #11277: why not verify INT32_MAX on 2.7? +# Also take care of Issue #1202 here +class ChecksumBigBufferTestCase(unittest.TestCase): +@unittest.skipUnless(mmap, mmap() is not available.) +def test_big_buffer(self): +if sys.platform[:3] == 'win' or sys.platform == 'darwin': +requires('largefile', +'test requires %s bytes and a long time to run' % str(_2G+2)) +try: +with open(TESTFN, wb+) as f: +f.seek(_2G-2) +f.write(asdf) +f.flush() +try: +m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) +self.assertEqual(zlib.crc32(m), -2072986226) +self.assertEqual(zlib.adler32(m), -2072641121) +finally: +m.close() +except (IOError, OverflowError): +raise unittest.SkipTest(filesystem doesn't have largefile support) +finally: +unlink(TESTFN) + + class ExceptionTestCase(unittest.TestCase): # make sure we generate some expected errors def test_badlevel(self): @@ -546,8 +578,9 @@ def test_main(): -test_support.run_unittest( +run_unittest( ChecksumTestCase, +ChecksumBigBufferTestCase, ExceptionTestCase, CompressTestCase, CompressObjectTestCase ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file21673/11277.zsum32.c ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com