[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment:
So sorry that i'm stressing this, hopefully it's the final message.
Apples iterative kernel-update strategy resulted in these versions:
14:02 ~/tmp $ /usr/sbin/sysctl kern.version
kern.version: Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011;
root:xnu-1504.15.3~1/RELEASE_I386
14:02 ~/tmp $ gcc -o zt osxversion.c -framework CoreServices
14:03 ~/tmp $ ./zt
OS X version: 10.6.8
apple_osx_needs_fullsync: -1
I.e. the new patch uses 10.7.0 or =10.6.8 to avoid that
FULLFSYNC disaster (even slower than the Macrohard memory
allocator during Wintel partnership!), and we end up as:
14:03 ~/src/cpython $ ./python.exe -E -Wd -m test -r -w -uall test_mmap
Using random seed 8466468
[1/1] test_mmap
1 test OK.
P.S.: i still have no idea how to do '-framework CoreServices'
regulary. Like i've said in #11046 i never used GNU Autoconf/M4,
sorry. You know. Maybe the version check should be moved
somewhere else and simply be exported, even replacing the stuff
from platform.py? I don't know. Bye.
--
Ciao, Steffen
sdaoden(*)(gmail.com)
() ascii ribbon campaign - against html e-mail
/\ www.asciiribbon.org - against proprietary attachments
--
Added file: http://bugs.python.org/file22593/11277.apple-fix-3.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11277
___diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst
--- a/Doc/library/mmap.rst
+++ b/Doc/library/mmap.rst
@@ -88,7 +88,8 @@
To ensure validity of the created memory mapping the file specified
by the descriptor *fileno* is internally automatically synchronized
- with physical backing store on Mac OS X and OpenVMS.
+ with physical backing store on operating systems where this is
+ necessary, e.g. OpenVMS, and some buggy versions of Mac OS X.
This example shows a simple way of using :class:`mmap`::
diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c
--- a/Modules/mmapmodule.c
+++ b/Modules/mmapmodule.c
@@ -25,6 +25,8 @@
#define UNIX
# ifdef __APPLE__
# include fcntl.h
+
+# include CoreServices/CoreServices.h
# endif
#endif
@@ -65,6 +67,44 @@
#define my_getpagesize getpagesize
#endif
+# ifdef __APPLE__
+static void
+apple_osx_needs_fullsync(long *use_fullsync)
+{
+/* Issue #11277: mmap(2) bug with 32 bit sparse files.
+ * Apple fixed the bug before announcement of OS X Lion, but since we
+ * need to handle buggy versions, perform a once-only check to see if the
+ * running kernel requires the expensive sync. Fixed in 10.6.8, 10.7++.
+ * 0: F_FULLSYNC is required, 0: kernel has mmap(2) bug fixed */
+SInt32 ver;
+*use_fullsync = 1;
+
+if (Gestalt(gestaltSystemVersion, ver) != noErr)
+goto jleave;
+/* SystemVersion(Major|Minor|BugFix) available at all? */
+if (ver 0x1040)
+goto jleave;
+if (Gestalt(gestaltSystemVersionMajor, ver) != noErr)
+goto jleave;
+if (ver 10)
+goto jgood;
+if (Gestalt(gestaltSystemVersionMinor, ver) != noErr)
+goto jleave;
+if (ver = 7)
+goto jgood;
+if (ver 6)
+goto jleave;
+if (Gestalt(gestaltSystemVersionBugFix, ver) != noErr)
+goto jleave;
+if (ver 8)
+goto jleave;
+jgood:
+*use_fullsync = -1;
+jleave:
+return;
+}
+# endif /* __APPLE__ */
+
#endif /* UNIX */
#include string.h
@@ -1150,8 +1190,14 @@
#ifdef __APPLE__
/* Issue #11277: fsync(2) is not enough on OS X - a special, OS X specific
fcntl(2) is necessary to force DISKSYNC and get around mmap(2) bug */
-if (fd != -1)
-(void)fcntl(fd, F_FULLFSYNC);
+if (fd != -1) {
+/* (GIL protected) */
+static long use_fullsync /*= 0*/;
+if (!use_fullsync)
+apple_osx_needs_fullsync(use_fullsync);
+if (use_fullsync 0)
+(void)fcntl(fd, F_FULLFSYNC);
+}
#endif
#ifdef HAVE_FSTAT
# ifdef __VMS
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file22281/11277.apple-fix-2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: @ Ronald Oussoren wrote: if major = 10: # We're on OSX 10.6 or earlier enableWorkaround() (You sound as if you participate in an interesting audiophonic event. 27 imac's are indeed great recording studio hardware. But no Coffee Shops in California - br.) -- Ciao, Steffen sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Ronald Oussoren ronaldousso...@mac.com added the comment: steffen: I have no idea what you are trying to say in your last message. Could you please try to stay on topic. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment:
Ok, this patch could be used.
*Unless* the code is not protected by the GIL.
- Gestalt usage is a bit more complicated according to
http://www.cocoadev.com/index.pl?DeterminingOSVersion
unless Python only supports OS X 10.4 and later.
(And platform.py correctly states that in _mac_ver_gestalt(),
but see below.)
- Due to usage of Gestalt, '-framework CoreServices' must be
linked against mmapmodule.c.
The Python configuration stuff is interesting for me, i managed
compilation by adding the line
mmap mmapmodule.c -framework CoreServices
to Modules/Setup, but i guess it's only OS X which is happy
about that.
platform.py: _mac_ver_xml() should be dropped entirely according
to one of Ned Deily's links (never officially supported), and
_mac_ver_gestalt() obviously never executed because i guess it
would fail due to versioninfo. Unless i missed something.
By the way: where do you get the info from? sys1, sys2,
sys3? Cannot find it anywhere, only the long names, e.g.
gestaltSystemVersionXy.
Note that i've mailed Apple. I did not pay 99$ or even 249$, so
i don't know if there will be a response.
--
Ciao, Steffen
sdaoden(*)(gmail.com)
() ascii ribbon campaign - against html e-mail
/\ www.asciiribbon.org - against proprietary attachments
--
Added file: http://bugs.python.org/file22281/11277.apple-fix-2.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11277
___diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst
--- a/Doc/library/mmap.rst
+++ b/Doc/library/mmap.rst
@@ -88,7 +88,8 @@
To ensure validity of the created memory mapping the file specified
by the descriptor *fileno* is internally automatically synchronized
- with physical backing store on Mac OS X and OpenVMS.
+ with physical backing store on operating systems where this is
+ necessary, e.g. OpenVMS, and some buggy versions of Mac OS X.
This example shows a simple way of using :class:`mmap`::
diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c
--- a/Modules/mmapmodule.c
+++ b/Modules/mmapmodule.c
@@ -25,6 +25,8 @@
#define UNIX
# ifdef __APPLE__
# include fcntl.h
+
+# include CoreServices/CoreServices.h
# endif
#endif
@@ -65,6 +67,39 @@
#define my_getpagesize getpagesize
#endif
+# ifdef __APPLE__
+static void
+apple_osx_needs_fullsync(long *use_fullsync)
+{
+/* Issue #11277: mmap(2) bug with 32 bit sparse files.
+ * Apple fixed the bug before announcement of OS X Lion, but since we
+ * need to handle buggy versions, perform a once-only check to see if the
+ * running kernel requires the expensive sync.
+ * 0: F_FULLSYNC is required, 0: kernel has mmap(2) bug fixed */
+SInt32 ver;
+
+*use_fullsync = 1;
+if (Gestalt(gestaltSystemVersion, ver) != noErr ver = 0x1040) {
+if (Gestalt(gestaltSystemVersionMajor, ver) != noErr)
+goto jleave;
+if (ver 10) {
+*use_fullsync = -1;
+goto jleave;
+}
+
+if (Gestalt(gestaltSystemVersionMinor, ver) != noErr)
+goto jleave;
+if (ver = 7) {
+*use_fullsync = -1;
+goto jleave;
+}
+}
+
+jleave:
+return;
+}
+# endif /* __APPLE__ */
+
#endif /* UNIX */
#include string.h
@@ -1128,8 +1163,14 @@
#ifdef __APPLE__
/* Issue #11277: fsync(2) is not enough on OS X - a special, OS X specific
fcntl(2) is necessary to force DISKSYNC and get around mmap(2) bug */
-if (fd != -1)
-(void)fcntl(fd, F_FULLFSYNC);
+if (fd != -1) {
+/* (GIL protected) */
+static long use_fullsync /*= 0*/;
+if (!use_fullsync)
+apple_osx_needs_fullsync(use_fullsync);
+if (use_fullsync 0)
+(void)fcntl(fd, F_FULLFSYNC);
+}
#endif
#ifdef HAVE_FSTAT
# ifdef __VMS
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file22273/11277.apple-fix.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Ronald Oussoren ronaldousso...@mac.com added the comment:
Steffen: _mac_ver_xml should not be dropped, it is a perfectly fine way to
determine the system version. Discussing it is also off-topic for this issue,
please keep the discussion focussed.
Wrt. mailing Apple: I wouldn't expect and answer. Is there something specific
you want to know? I'm currently at WWDC and might be able to ask the question
at one of the labs (where Apple's engineers hang out).
If it is really necessary to check for the OS version to enable the
OSX-specific bugfix it is possible to look at the uname information instead of
using gestalt. In particular something simular to this Python code:
v = os.uname()[2]
major = int(v.split('.')[0])
if major = 10:
# We're on OSX 10.6 or earlier
enableWorkaround()
This tests the kernel version instead of the system version, but until now the
major version of the kernel has increased with every major release of the OS
and I have no reason to suspect that Lion will be any different.
BTW2: OSX 10.7 is not released yet and should not be discussed in public fora,
as you should know if you have legal access.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11277
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Aehm, note that Apple has fixed the mmap(2) bug!! I'm still surprised and can't really believe it, but it's true! Just in case you're interested, i'll apply an updated patch. Maybe Ned Deily should have a look at the version check, which does not apply yet, but i don't know any other way to perform exact version checking. (Using 10.6.7 is not enough, it must be 10.7.0; uname -a yet reports that all through, but no CPP symbol does!?) -- Added file: http://bugs.python.org/file22273/11277.apple-fix.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Ned Deily n...@acm.org added the comment: Thanks for the update. Since the fix will be in a future version of OS X 10.7 Lion, and which has not been released yet, so it is not appropriate to change mmap until there has been an opportunity to test it. But even then, we would need to be careful about adding a compile-time test as OS X binaries are often built to be compatible for a range of operating system version so avoid adding compilation conditionals unless really necessary. If after 10.7 is released and someone is able to test that it works as expected, the standard way to support it would be to use the Apple-supplied availability macros to test for the minimum supported OS level of the build assuming it makes enough of a performance difference to bother to do so: http://developer.apple.com/library/mac/#technotes/tn2064/_index.html (Modules/_ctypes/darwin/dlfcn_simple.c is one of the few that has this kind of test.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: @ Ned Deily rep...@bugs.python.org wrote (2011-06-07 19:43+0200): Thanks for the update. Since the fix will be in a future version of OS X 10.7 Lion, and which has not been released yet, so it is not appropriate to change mmap until there has been an opportunity to test it. It's really working fine. That i see that day! (Not that they start to fix the CoreAudio crashes...) But even then, we would need to be careful about adding a compile-time test as OS X binaries are often built to be compatible for a range of operating system version so avoid adding compilation conditionals unless really necessary. If after 10.7 is released and someone is able to test that it works as expected, the standard way to support it would be to use the Apple-supplied availability macros to test for the minimum supported OS level of the build assuming it makes enough of a performance difference to bother to do so Of course it only moves the delay from before mmap(2) to after close(2). Well, i don't know, if hardcoding is not an option, a dynamic sysctl(2) lookup may do: kern.version = Darwin Kernel Version 10.7.0: Sat Jan 29 15:17:16 PST 2011 This is obviously not the right one. :) -- Ciao, Steffen sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
STINNER Victor victor.stin...@haypocalc.com added the comment:
Yes, you should check the Mac OS X version at runtime (as you should check the
Linux kernel at runtime). platform.mac_ver() uses something like:
sysv = _gestalt.gestalt('sysv')
if sysv:
major = (sysv 0xFF00) 8
minor = (sysv 0x00F0) 4
patch = (sysv 0x000F)
Note: patch is not reliable with 'sysv', you have to use ('sys1','sys2','sys3').
So if you would like to check that you have Mac OS 10.7 or later, you can do
something like:
sysv = _gestalt.gestalt('sysv')
__MAC_10_7 = (sysv and (sysv 4) = 0x0a7)
In C, it should be something like:
---
const OSType SYSV = 0x73797376U; /* 'sysv' in big endian */
SInt32 response;
OSErr iErr;
iErr = Gestalt(SYSV, response);
if (iErr == 0 (response 4) = 0x0a7)
/* have Mac OS = 10.7 */
---
I'm not sure of 0x73797376, I used hex(struct.unpack('!I', 'sysv')[0]).
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11277
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Ned Deily n...@acm.org added the comment:
Victor, please do not use magic constants like that in C. The symbolic values
are available in include files:
#include CoreServices/CoreServices.h
SInt32 major = 0;
SInt32 minor = 0;
Gestalt(gestaltSystemVersionMajor, major);
Gestalt(gestaltSystemVersionMinor, minor);
if ((major == 10 minor = 7) || major = 11) { ... }
(See, for instance, http://www.cocoadev.com/index.pl?DeterminingOSVersion and
http://stackoverflow.com/questions/2115373/os-version-checking-in-cocoa. The
code in platform and _gestalt.c could stand to be updated at some point.)
But, again, mmap should *not* be changed until 10.7 has been released and the
Apple fix is verified and only if it makes sense to add the additional
complexity.
--
nosy: +ronaldoussoren
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11277
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
STINNER Victor victor.stin...@haypocalc.com added the comment: @haypo, @pitrou: Are there any objections to removing test_big_buffer() from Lib/test/test_zlib.py? I now agree Antoine: the test is useless. It can be removed today. About mmap: add a new test for this issue (mmap on Mac OS X and F_FULLSYNC) is a good idea. I suppose that we will need to backport the F_FULLSYNC fix too. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Roundup Robot devnull@devnull added the comment: New changeset 201dcfc56e86 by Nadeem Vawda in branch '2.7': Issue #11277: Remove useless test from test_zlib. http://hg.python.org/cpython/rev/201dcfc56e86 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Roundup Robot devnull@devnull added the comment: New changeset d5d4f2967879 by Nadeem Vawda in branch '3.1': Issue #11277: Add tests for mmap crash when using large sparse files on OS X. http://hg.python.org/cpython/rev/d5d4f2967879 New changeset e447a68742e7 by Nadeem Vawda in branch '3.2': Merge: #11277: Add tests for mmap crash when using large sparse files on OS X. http://hg.python.org/cpython/rev/e447a68742e7 New changeset bc13badf10a1 by Nadeem Vawda in branch 'default': Merge: #11277: Add tests for mmap crash when using large sparse files on OS X. http://hg.python.org/cpython/rev/bc13badf10a1 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Roundup Robot devnull@devnull added the comment: New changeset 8d27d2b22394 by Nadeem Vawda in branch '2.7': Issue #11277: Add tests for mmap crash when using large sparse files on OS X. http://hg.python.org/cpython/rev/8d27d2b22394 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: @Nadeem: note that the committed versions of the tests would not show up the Mac OS X mmap() bug AFAIK, because there is an intermediate .close() of the file to be mmapped. The OS X bug is that the VMS/VFS interaction fails to provide a valid memory region for pages which are not yet physically present on disc - i.e. there is no true sparse file support as on Linux, which simply uses references to a single COW zero page. (I've not tried it out for real yet, but i'm foolish like a prowd cock, so i've looked at the changeset :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: (Of course this may also be intentional, say. But then i would vote against it :), because it's better the tests bring out errors than end-user apps.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Roundup Robot devnull@devnull added the comment: New changeset 9b9f0de19684 by Nadeem Vawda in branch '2.7': Issue #11277: Fix tests - crash will not trigger if the file is closed and reopened. http://hg.python.org/cpython/rev/9b9f0de19684 New changeset b112c72f8c01 by Nadeem Vawda in branch '3.1': Issue #11277: Fix tests - crash will not trigger if the file is closed and reopened. http://hg.python.org/cpython/rev/b112c72f8c01 New changeset a9da17fcb564 by Nadeem Vawda in branch '3.2': Merge: #11277: Fix tests - crash will not trigger if the file is closed and reopened. http://hg.python.org/cpython/rev/a9da17fcb564 New changeset b3a94906c4a0 by Nadeem Vawda in branch 'default': Merge: #11277: Fix tests - crash will not trigger if the file is closed and reopened. http://hg.python.org/cpython/rev/b3a94906c4a0 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Nadeem Vawda nadeem.va...@gmail.com added the comment: sdaoden @Nadeem: note that the committed versions of the tests would not sdaoden show up the Mac OS X mmap() bug AFAIK, because there is an sdaoden intermediate .close() of the file to be mmapped. Thanks for catching that. Should be fixed now. haypo I now agree Antoine: the test is useless. It can be removed today. haypo haypo About mmap: add a new test for this issue (mmap on Mac OS X and haypo F_FULLSYNC) is a good idea. Done and done. haypo I suppose that we will need to backport the F_FULLSYNC fix too. It has already been backported, as changeset 618c3e971e80. -- resolution: - fixed stage: needs patch - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment:
On Fri, 6 May 2011 02:54:07 +0200, Nadeem Vawda wrote:
I think so. [.]
it turns out that the OS X sparsefile crash is also covered by
LargeMmapTests.test_large_offset() in test_mmap [!!!]. [.]
So i followed your suggestion and did not do something on zlib no
more. Even if that means that there is no test which checksums an
entire superlarge mmap() region.
Instead i've changed/added test cases in test_mmap.py:
- Removed all context-manager usage from LargeMmapTests().
This feature has been introduced in 3.2 and is already tested
elsewhere. Like this the test is almost identical on 2.7 and 3.x.
- I've dropped _working_largefile(). This creates a useless large
file only to unlink it directly. Instead the necessary try:catch:
is done directly in the tests.
- (Directly testing after .flush() without reopening the file.)
- These new tests don't run on 32 bit.
May the juice be with you
--
Added file: http://bugs.python.org/file21909/11277-test_mmap.1.py
Added file: http://bugs.python.org/file21910/11277-test_mmap-27.1.py
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11277
___diff --git a/Lib/test/test_mmap.py b/Lib/test/test_mmap.py
--- a/Lib/test/test_mmap.py
+++ b/Lib/test/test_mmap.py
@@ -1,4 +1,5 @@
-from test.support import TESTFN, run_unittest, import_module, unlink, requires
+from test.support import TESTFN, run_unittest, import_module, unlink
+from test.support import requires, _4G
import unittest
import os
import re
@@ -662,44 +663,87 @@
def tearDown(self):
unlink(TESTFN)
-def _working_largefile(self):
-# Only run if the current filesystem supports large files.
-f = open(TESTFN, 'wb', buffering=0)
+def _test_splice(self, f, i):
+# Test splicing with pages around critical values in respect to
+# memory management
+# Issue 11277: does mmap() force materialization of backing store?
+m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
try:
-f.seek(0x8001)
-f.write(b'x')
-f.flush()
-except (IOError, OverflowError):
-raise unittest.SkipTest(filesystem does not have largefile
support)
+# Memory page before xy
+self.assertEqual(m[i+0:i+2], b' ')
+# Memory page after xy
+self.assertEqual(m[i+10:i+12], b' ')
+# Cross pages
+self.assertEqual(m[i+2:i+10], b'DEARdear')
finally:
-f.close()
-unlink(TESTFN)
+m.close()
-def test_large_offset(self):
+def _test_subscr(self, f, idx, expect):
+# Test subscript for critical values like INT32_MAX, UINT32_MAX
+m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
+try:
+self.assertEqual(m[idx], expect)
+finally:
+m.close()
+
+@unittest.skipUnless(sys.maxsize _4G, Can't run on a 32-bit system.)
+def test_around_32bit_sbitflip(self):
+start = 0x7FFA
if sys.platform[:3] == 'win' or sys.platform == 'darwin':
requires('largefile',
-'test requires %s bytes and a long time to run' %
str(0x18000))
-self._working_largefile()
-with open(TESTFN, 'wb') as f:
-f.seek(0x14FFF)
-f.write(b )
+ 'test requires %s bytes and a long time to run' %
+ str(start+12))
+with open(TESTFN, 'w+b') as f:
+try:
+f.seek(start)
+f.write(b' DEARdear ')
+f.flush()
+except (IOError, OverflowError):
+raise unittest.SkipTest('filesystem does not have largefile '
+'support')
+self._test_splice(f, start)
+self._test_subscr(f, start+len(b' DEA'), ord(b'R'))
+self._test_subscr(f, start+len(b' DEARdea'), ord(b'r'))
+unlink(TESTFN)
-with open(TESTFN, 'rb') as f:
-with mmap.mmap(f.fileno(), 0, offset=0x14000,
access=mmap.ACCESS_READ) as m:
-self.assertEqual(m[0xFFF], 32)
+@unittest.skipUnless(sys.maxsize _4G, Can't run on a 32-bit system.)
+def test_around_32bit_excess(self):
+start = 0xFFFA
+if sys.platform[:3] == 'win' or sys.platform == 'darwin':
+requires('largefile',
+ 'test requires %s bytes and a long time to run' %
+ str(start+12))
+with open(TESTFN, 'w+b') as f:
+try:
+f.seek(start)
+f.write(b' DEARdear ')
+f.flush()
+except (IOError, OverflowError):
+raise unittest.SkipTest('filesystem does not have largefile '
+
[issue11277] Crash with mmap and sparse files on Mac OS X
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file21869/11277-27.2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file21885/11277-27.3.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Nadeem Vawda nadeem.va...@gmail.com added the comment: Thanks for the tests; I'll review and commit them tomorrow morning. Even if that means that there is no test which checksums an entire superlarge mmap() region. Bear in mind that the test is only to be removed from 2.7; it will still be present in the 3.* branches, where crc32() and adler32() actually can accept such large inputs. @haypo, @pitrou: Are there any objections to removing test_big_buffer() from Lib/test/test_zlib.py? If not, I think we can close this issue once that and the additional mmap tests are committed. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
STINNER Victor victor.stin...@haypocalc.com added the comment: @sdaoden(, @pitrou): Antoine proposes to skip the zlib big buffer (1 GB) test on 32 bits system. What do you think? On 64 bits system, we check a buffer of 2 GB-1 byte (0x7FFF bytes). Is the test useful or not? What do we test? Can you check if the test crashs on Mac OS X on a 32 bits system (1 GB buffer) if you disable F_FULLFSYNC in mmapmodule.c? Same question on a 64 bits system (2 GB-1 byte buffer)? The most important test if to test crc32 adler32 with a buffer bigger than 4 GB, but we cannot write such test in Python 2.7 because the zlib module stores buffer sizes into int variables. So the big buffer test of Python 2.7 test_zlib is maybe just useful (on 32 and 64 bits). Can we just remove the test? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: @haypo: trouble, trouble on the dev-list, as i've seen currently. Sorry, sorry. (Cannot subscribe, my DynIP's are often blacklisted ;) Of course my comments were completely wrong, as Ethan has pointed out correctly. This is all s**t. These are mmap(2) related issues and should be tested in Lib/test/test_mmap.py. However that does not use with open: create sparse file materialize yet so that the Pear OS X sparsefile bug doesn't show up. In fact it doesn't do a full beam-me-up test at all yet? Is the test useful or not? What do we test? We do test that mmap.mmap materializes a buffer which can be accessed (readonly) from [0]..[len-1]. And that the checksums that zlib produces for that buffer are correct. Unfortunately we cannot test 0x8000+ no more because Python prevents that such a buffer can be used - that's a shame. Maybe we could test 0x7FFF*2 aka 0xfffe in two iterations. Can you check if the test crashs on Mac OS X on a 32 bits system (1 GB buffer) if you disable F_FULLFSYNC in mmapmodule.c? Same question on a 64 bits system (2 GB-1 byte buffer)? Aeh - F_FULLFSYNC was not yet committed at that time in 2.7. Can we just remove the test? If i - choke! - need to write tests, i try to catch corner cases. The corner cases would be 0,MAX_LEN(-1) and some (rather pseudo) random values around these and maybe some in the middle. (Plus some invalid inputs.) Can we remove it? I would keep it, Apple is preparing the next major release (uname -a yet states 10.7.0 even though it's 10.6.7), and maybe then mmap() will fail for 0xDEADBEEF. Who will be the one which detects that otherwise?? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: In fact i like my idea of using iterations. I have some time tomorrow, so if nobody complains until then, i write diffs for the tests of 3.x and 2.7 with these updates: - Two different target sizes: 1. 0x + x (7) 2. 0x7FFF + x (7) - On 32 bit systems, use iterations on a potentially safe buffer size. I think 0x4000 a.k.a 1024*1024*1024 is affordable, but 512 MB are probably more safe? I'll make that a variable. - The string will be 'DeadAffe' (8). - The last 4 bytes of the string will always be read on their own (just in case the large buffer sizes irritated something down the path). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Nadeem Vawda nadeem.va...@gmail.com added the comment: haypo Can we just remove the test? I think so. The test was originally intended to catch the case where crc32() or adler32() would get a buffer of =4GB, and then silently truncate the size and produce an incorrect result (issue10276). However, 2.7's zlib doesn't define PY_SSIZE_T_CLEAN, so passing in a buffer of =2GB raises an exception. So the condition that it was testing for can't happen in 2.7. sdaoden Can we remove it? I would keep it, Apple is preparing the next sdaoden major release (uname -a yet states 10.7.0 even though it's sdaoden 10.6.7), and maybe then mmap() will fail for 0xDEADBEEF. sdaoden Who will be the one which detects that otherwise?? I initially thought the same thing, but it turns out that the OS X sparsefile crash is also covered by LargeMmapTests.test_large_offset() in test_mmap. That test had also been failing sporadically before the F_FULLSYNC patch was committed (see issue11779). So keeping this test around would be redundant. sdaoden Unfortunately we cannot test 0x8000+ no more because sdaoden Python prevents that such a buffer can be used - that's a shame. sdaoden Maybe we could test 0x7FFF*2 aka 0xfffe in two iterations. That wouldn't accomplish the same thing. The point of the test is to pick up truncation issues that occur when you pass in a big buffer. These issues won't show up if you split the data up into smaller pieces. And in any case, they can't happen at all in 2.7, because the functions don't accept big buffers in the first place ;) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
STINNER Victor victor.stin...@haypocalc.com added the comment: Reopen, test_zlib fails with Python 2.7 on Windows: == ERROR: test_big_buffer (test.test_zlib.ChecksumBigBufferTestCase) -- Traceback (most recent call last): File D:\cygwin\home\db3l\buildarea\2.7.bolen-windows\build\lib\test\test_zlib.py, line 91, in test_big_buffer m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) WindowsError: [Error 87] The parameter is incorrect http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%202.7/builds/854/steps/test/logs/stdio -- resolution: fixed - status: closed - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Roundup Robot devnull@devnull added the comment: New changeset 1ef2a7319849 by Victor Stinner in branch '2.7': Issue #11277: fix issue number in a test_zlib comment http://hg.python.org/cpython/rev/1ef2a7319849 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
STINNER Victor victor.stin...@haypocalc.com added the comment:
x86 debian parallel 2.7, x86 Ubuntu Shared 2.7 and x86 Tiger 2.7 fail
with mmap.error('[Errno 12] Cannot allocate memory').
http://www.python.org/dev/buildbot/all/builders/x86%20Ubuntu%20Shared%202.7/builds/866/steps/test/logs/stdio
http://www.python.org/dev/buildbot/all/builders/x86%20Tiger%202.7/builds/776/steps/test/logs/stdio
http://www.python.org/dev/buildbot/all/builders/x86%20debian%20parallel%202.7/builds/739/steps/test/logs/stdio
==
ERROR: test_big_buffer (test.test_zlib.ChecksumBigBufferTestCase)
--
Traceback (most recent call last):
File /srv/buildbot/buildarea/2.7.bolen-ubuntu/build/Lib/test/test_zlib.py,
line 91, in test_big_buffer
m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
error: [Errno 12] Cannot allocate memory
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11277
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment:
error: [Errno 12] Cannot allocate memory
@haypo: Well i told you i have no idea. These bots are 32 bit?
I'll attach 11277-27.3.diff which does @skipUnless(not 32 bit).
Note i'll test against _4G - does this work (on 32 bit and in
Python)? A pity that Python does not offer a 'condition is
always true due to datatype storage restriction' check?!
And i don't think it makes sense to test a _1GB mmap on 32 bit at
all (but at least address space shouldn't exhaust for that).
So, sorry, also for the two bugs in that two-liner, but very
especially the 'm' case.
--
Added file: http://bugs.python.org/file21885/11277-27.3.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11277
___diff --git a/Lib/test/test_zlib.py b/Lib/test/test_zlib.py
--- a/Lib/test/test_zlib.py
+++ b/Lib/test/test_zlib.py
@@ -2,7 +2,7 @@
from test.test_support import TESTFN, run_unittest, import_module, unlink,
requires
import binascii
import random
-from test.test_support import precisionbigmemtest, _1G
+from test.test_support import precisionbigmemtest, _1G, _2G, _4G
import sys
try:
@@ -75,17 +75,16 @@
# Issue #11277 - check that inputs of 2 GB are handled correctly.
# Be aware of issues #1202, #8650, #8651 and #10276
class ChecksumBigBufferTestCase(unittest.TestCase):
-int_max = 0x7FFF
-
+@unittest.skipUnless(sys.maxsize _4G, Can't run on a 32-bit system.)
@unittest.skipUnless(mmap, mmap() is not available.)
def test_big_buffer(self):
if sys.platform[:3] == 'win' or sys.platform == 'darwin':
requires('largefile',
'test requires %s bytes and a long time to run' %
- str(self.int_max))
+ str(_2G -1))
try:
with open(TESTFN, wb+) as f:
-f.seek(self.int_max-4)
+f.seek(_2G -1-4)
f.write(asdf)
f.flush()
m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Roundup Robot devnull@devnull added the comment: New changeset 7f3cab59ef3e by Victor Stinner in branch '2.7': Issue #11277: test_zlib tests a buffer of 1 GB on 32 bits http://hg.python.org/cpython/rev/7f3cab59ef3e -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Roundup Robot devnull@devnull added the comment: New changeset e6a4deb84e47 by Victor Stinner in branch '2.7': Issue #11277: oops, fix checksum values of test_zlib on 32 bits http://hg.python.org/cpython/rev/e6a4deb84e47 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: @haypo: Oh. Not: if sys.maxsize _4G: # (64 bits system) crc32() and adler32() stores the buffer size into an # int, the maximum filesize is INT_MAX (0x7FFF) filesize = 0x7FFF crc_res = 0x709418e7 adler_res = -2072837729 else: # (32 bits system) On a 32 bits OS, a process cannot usually address # more than 2 GB, so test only 1 GB filesize = _1G crc_res = 0x2b09ee11 adler_res = -1002962529 self.assertEqual(zlib.crc32(m), self.crc_res) self.assertEqual(zlib.adler32(m), self.adler_res) I'm not that fast. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment:
Should we fix Python 2.7?
- backport issue #8651
- use PY_SSIZE_T_CLEAN in zlibmodule.c
I really thought about this over night.
I'm a C programmer and thus:
- Produce no bugs
- If you've produced a bug, fix it at once
- If you've fixed a bug, scream out loud BUGFIX! -
or at least incorporate the patch in the very next patch release
But i have no experience with maintaining a scripting language.
My survey of something like this spans about three months now.
And if even such a heavy known bug as #1202 survives at least two
minor releases (2.6 and 2.7) without being fixed, then maybe no
more effort should be put into 2.7 at all.
11277-27.1.diff contains # Issue #10276 - check that inputs
=4GB are handled correctly.. I don't understand this comment
because the test uses a buffer of 2 GB + 2 bytes.
How is it possible to pass a buffer of 2 GB+2 bytes to crc32(),
whereas it stores the size into an int. The maximum size is
INT_MAX which is 2 GB-1 byte. It looks like the i format of
PyArg_ParseTuple() doesn't check for integer overflow = issue
#8651. This issue was fixed in 3.1, 3.2 and 3.3, but not in
Python 2
11277-27.2.diff uses INT_MAX and thus avoids any such pitfall.
Maybe it brings up memory mapping errors somewhere which i surely
would try fix everywhere i can.
--
Added file: http://bugs.python.org/file21869/11277-27.2.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11277
___diff --git a/Lib/test/test_zlib.py b/Lib/test/test_zlib.py
--- a/Lib/test/test_zlib.py
+++ b/Lib/test/test_zlib.py
@@ -1,10 +1,16 @@
import unittest
-from test import test_support
+from test.test_support import TESTFN, run_unittest, import_module, unlink,
requires
import binascii
import random
from test.test_support import precisionbigmemtest, _1G
+import sys
-zlib = test_support.import_module('zlib')
+try:
+import mmap
+except ImportError:
+mmap = None
+
+zlib = import_module('zlib')
class ChecksumTestCase(unittest.TestCase):
@@ -66,6 +72,34 @@
zlib.crc32('spam', (2**31)))
+# Backport to 2.7 due to Issue #11277: why not also verify INT32_MAX on 2.7?
+# Be aware of issues #1202, #8650, #8651 and #10276
+class ChecksumBigBufferTestCase(unittest.TestCase):
+int_max = 0x7FFF
+
+@unittest.skipUnless(mmap, mmap() is not available.)
+def test_big_buffer(self):
+if sys.platform[:3] == 'win' or sys.platform == 'darwin':
+requires('largefile',
+ 'test requires %s bytes and a long time to run' %
+ str(self.int_max))
+try:
+with open(TESTFN, wb+) as f:
+f.seek(self.int_max-4)
+f.write(asdf)
+f.flush()
+try:
+m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
+self.assertEqual(zlib.crc32(m), 0x709418e7)
+self.assertEqual(zlib.adler32(m), -2072837729)
+finally:
+m.close()
+except (IOError, OverflowError):
+raise unittest.SkipTest(filesystem doesn't have largefile
support)
+finally:
+unlink(TESTFN)
+
+
class ExceptionTestCase(unittest.TestCase):
# make sure we generate some expected errors
def test_badlevel(self):
@@ -546,8 +580,9 @@
def test_main():
-test_support.run_unittest(
+run_unittest(
ChecksumTestCase,
+ChecksumBigBufferTestCase,
ExceptionTestCase,
CompressTestCase,
CompressObjectTestCase
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file21855/11277-27.1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Roundup Robot devnull@devnull added the comment: New changeset 618c3e971e80 by Victor Stinner in branch '2.7': (Merge 3.1) Issue #11277: mmap.mmap() calls fcntl(fd, F_FULLFSYNC) on Mac OS X http://hg.python.org/cpython/rev/618c3e971e80 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
STINNER Victor victor.stin...@haypocalc.com added the comment: I commited mmap fix for Mac OS X, crc test on 2 GB file, and issue #8651 fix into Python 2.7. Use PY_SSIZE_T_CLEAN in zlibmodule.c is a new feature. I don't want to implement it, I don't need it, and I don't feel confortable in zlibmodule.c. Open a new issue if you want it. I think that we are done with issue. If you see buildbots failures, reopen it. Thanks again Steffen for your diagnosis and fixes. -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment:
On Mon, 2 May 2011 01:22:41 +0200, STINNER Victor rep...@bugs.python.org
wrote:
@sdaoden: Can you try on Python 2.7?
@haypo: Python 2.7 is absolute horror.
But i tried and produced a (terrible - i don't know the test
framework and that test_support stuff seems to have been changed
a lot since 2.7) 2 gigabyte+ big buffer test for 2.7.
(Of course: even though Python uses int, ZLib uses uInt.)
It took some time because i fell over #1202 from 2007 unprepared.
The (nasty) test works quite well on Apple, which is not such
a big surprise, because Apple's OS X is especially designed for
artists which need to work on large files, like video+ cutters,
sound designers with sample databases etc., so i would be
terribly disappointed if that wouldn't work! Apple even
propagandize OS X for, and makes money with that very application
task - i really couldn't understand your doubts here.
--
Added file: http://bugs.python.org/file21855/11277-27.1.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11277
___diff --git a/Lib/test/test_zlib.py b/Lib/test/test_zlib.py
--- a/Lib/test/test_zlib.py
+++ b/Lib/test/test_zlib.py
@@ -1,10 +1,16 @@
import unittest
-from test import test_support
+from test.test_support import TESTFN, run_unittest, import_module, unlink,
requires
+from test.test_support import precisionbigmemtest, _1G, _2G
import binascii
import random
-from test.test_support import precisionbigmemtest, _1G
+import sys
-zlib = test_support.import_module('zlib')
+try:
+import mmap
+except ImportError:
+mmap = None
+
+zlib = import_module('zlib')
class ChecksumTestCase(unittest.TestCase):
@@ -66,6 +72,32 @@
zlib.crc32('spam', (2**31)))
+# Issue #10276 - check that inputs =4GB are handled correctly.
+# Backport to 2.7 due to Issue #11277: why not verify INT32_MAX on 2.7?
+# Also take care of Issue #1202 here
+class ChecksumBigBufferTestCase(unittest.TestCase):
+@unittest.skipUnless(mmap, mmap() is not available.)
+def test_big_buffer(self):
+if sys.platform[:3] == 'win' or sys.platform == 'darwin':
+requires('largefile',
+'test requires %s bytes and a long time to run' % str(_2G+2))
+try:
+with open(TESTFN, wb+) as f:
+f.seek(_2G-2)
+f.write(asdf)
+f.flush()
+try:
+m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
+self.assertEqual(zlib.crc32(m), -2072986226)
+self.assertEqual(zlib.adler32(m), -2072641121)
+finally:
+m.close()
+except (IOError, OverflowError):
+raise unittest.SkipTest(filesystem doesn't have largefile
support)
+finally:
+unlink(TESTFN)
+
+
class ExceptionTestCase(unittest.TestCase):
# make sure we generate some expected errors
def test_badlevel(self):
@@ -546,8 +578,9 @@
def test_main():
-test_support.run_unittest(
+run_unittest(
ChecksumTestCase,
+ChecksumBigBufferTestCase,
ExceptionTestCase,
CompressTestCase,
CompressObjectTestCase
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file21673/11277.zsum32.c ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
STINNER Victor victor.stin...@haypocalc.com added the comment: @haypo: Python 2.7 is absolute horror. Oh, zlib doesn't use PY_SSIZE_T_CLEAN in Python 2.7. 11277-27.1.diff contains # Issue #10276 - check that inputs =4GB are handled correctly.. I don't understand this comment because the test uses a buffer of 2 GB + 2 bytes. How is it possible to pass a buffer of 2 GB+2 bytes to crc32(), whereas it stores the size into an int. The maximum size is INT_MAX which is 2 GB-1 byte. It looks like the i format of PyArg_ParseTuple() doesn't check for integer overflow = issue #8651. This issue was fixed in 3.1, 3.2 and 3.3, but not in Python 2. Should we fix Python 2.7? - backport issue #8651 - use PY_SSIZE_T_CLEAN in zlibmodule.c -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Roundup Robot devnull@devnull added the comment: New changeset cb464f8fb3a1 by Victor Stinner in branch '3.1': Issue #11277: mmap calls fcntl(fd, F_FULLFSYNC) on Mac OS X to get around a http://hg.python.org/cpython/rev/cb464f8fb3a1 New changeset e9d298376dde by Victor Stinner in branch '3.2': (Merge 3.1) Issue #11277: mmap.mmap() calls fcntl(fd, F_FULLFSYNC) on Mac OS X http://hg.python.org/cpython/rev/e9d298376dde New changeset d578fdc9b157 by Victor Stinner in branch 'default': (Merge 3.2) Issue #11277: mmap.mmap() calls fcntl(fd, F_FULLFSYNC) on Mac OS X http://hg.python.org/cpython/rev/d578fdc9b157 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
STINNER Victor victor.stin...@haypocalc.com added the comment: I am not able to check the fix, but the buildbots are :-) What should be done for Python 2.7? In Python 2.7, zlib.crc32() stores the buffer length into an int (so the maximum length is INT_MAX), and so test_zlib doesn't test a (sparse) file of 4 GB (ChecksumBigBufferTestCase). But I suppose that mmap bug can also occur with a file of 2 GB. @sdaoden: Can you try on Python 2.7? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: What do you think - i think this issue can really be closed now. I'll attach a final 11277.5.diff which has a less irritated and thus better understandable comment than .4.diff. I'll also drop .3 and .4. A lot of noise again 8| -- Added file: http://bugs.python.org/file21798/11277.5.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst --- a/Doc/library/mmap.rst +++ b/Doc/library/mmap.rst @@ -86,6 +86,10 @@ defaults to 0. *offset* must be a multiple of the PAGESIZE or ALLOCATIONGRANULARITY. + To ensure validity of the created memory mapping the file specified + by the descriptor *fileno* is internally automatically synchronized + with physical backing store on Mac OS X and OpenVMS. + This example shows a simple way of using :class:`mmap`:: import mmap diff --git a/Lib/test/test_zlib.py b/Lib/test/test_zlib.py --- a/Lib/test/test_zlib.py +++ b/Lib/test/test_zlib.py @@ -70,7 +70,7 @@ with open(support.TESTFN, wb+) as f: f.seek(_4G) f.write(basdf) -with open(support.TESTFN, rb) as f: +f.flush() self.mapping = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) def tearDown(self): diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c --- a/Modules/mmapmodule.c +++ b/Modules/mmapmodule.c @@ -23,6 +23,9 @@ #ifndef MS_WINDOWS #define UNIX +# ifdef __APPLE__ +# include fcntl.h +# endif #endif #ifdef MS_WINDOWS @@ -1122,6 +1125,12 @@ mmap invalid access parameter.); } +#ifdef __APPLE__ +/* Issue 11277: fsync(2) is not enough on OS X - a special, OS X specific + * fcntl(2) is necessary to force DISKSYNC and get around mmap(2) bug */ +if (fd != -1) +(void)fcntl(fd, F_FULLFSYNC); +#endif #ifdef HAVE_FSTAT # ifdef __VMS /* on OpenVMS we must ensure that all bytes are written to the file */ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file21715/11277.3.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Changes by Steffen Daode Nurpmeso sdao...@googlemail.com: Removed file: http://bugs.python.org/file21717/11277.4.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Changes by STINNER Victor victor.stin...@haypocalc.com: -- title: test_zlib.test_big_buffer crashes under BSD (Mac OS X and FreeBSD) - Crash with mmap and sparse files on Mac OS X ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
STINNER Victor victor.stin...@haypocalc.com added the comment: Oh, fcntl has already a F_FULLFSYNC constant, so we can use like fcntl.fcntl(fd, fcntl.F_FULLFSYNC) in Python. can you add a sentence in mmap doc to explain that mmap.mmap() does flush the file on Mac OS X and VMS? Hum, it does flush the file on VMS using fsync(), but on Mac OS X, it does just set F_FULLFSYNC flag using fcntl. It doesn't call fsync() explicitly. Does mmap() call fsync() implicitly? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
STINNER Victor victor.stin...@haypocalc.com added the comment: Oh, and can you add a comment explaining why F_FULLFSYNC is needed on Mac OS X in your patch? (If I understood correctly, it is needed to avoid crash with sparse files). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: Updated 11277.4.diff also includes mmap.rst update. (Maybe os.fsync() and os.sync() should be modified to really do that fcntl, too? I'll think i'll open an issue with patch soon.) -- Added file: http://bugs.python.org/file21717/11277.4.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst --- a/Doc/library/mmap.rst +++ b/Doc/library/mmap.rst @@ -86,6 +86,10 @@ defaults to 0. *offset* must be a multiple of the PAGESIZE or ALLOCATIONGRANULARITY. + To ensure validity of the created memory mapping the file specified + by the descriptor *fileno* is internally automatically synchronized + with physical backing store on Mac OS X and OpenVMS. + This example shows a simple way of using :class:`mmap`:: import mmap diff --git a/Lib/test/test_zlib.py b/Lib/test/test_zlib.py --- a/Lib/test/test_zlib.py +++ b/Lib/test/test_zlib.py @@ -70,7 +70,7 @@ with open(support.TESTFN, wb+) as f: f.seek(_4G) f.write(basdf) -with open(support.TESTFN, rb) as f: +f.flush() self.mapping = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) def tearDown(self): diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c --- a/Modules/mmapmodule.c +++ b/Modules/mmapmodule.c @@ -23,6 +23,9 @@ #ifndef MS_WINDOWS #define UNIX +# ifdef __APPLE__ +# include fcntl.h +# endif #endif #ifdef MS_WINDOWS @@ -1122,6 +1125,13 @@ mmap invalid access parameter.); } +#ifdef __APPLE__ +/* Issue 11277: due to lack of sparse file support on OS X synchronization + * with physical backing store is required to ensure validity of the mmap. + * Since fsync(2) acts like fdatasync(2) a special fcntl(2) is necessary */ +if (fd != -1) +(void)fcntl(fd, F_FULLFSYNC); +#endif #ifdef HAVE_FSTAT # ifdef __VMS /* on OpenVMS we must ensure that all bytes are written to the file */ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso sdao...@googlemail.com added the comment: (My last reply-mail changed the title. Fixing.) -- title: test_zlib.test_big_buffer crashes under BSD (Mac OS X and FreeBSD) - Crash with mmap and sparse files on Mac OS X ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
STINNER Victor victor.stin...@haypocalc.com added the comment: (My last reply-mail changed the title. Fixing.) Yeah, it's a common problem if you use the email interface :-/ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
