Re: [pypy-dev] Patches for Pypy under cygwin
On Mon, Apr 30, 2012 at 7:05 PM, Uwe F. Mayer uwe_f_ma...@yahoo.com wrote: In particular the work-around for the tm structure to get by without the tm_gmtoff and tm_zone fields may be of interest, as it allows compilation of the rctime module on systems that don't have it, such as some versions of SunOS. Interesting and timely. I am about to add limited tm_gmtoff/tm_zone support to CPython. See http://bugs.python.org/issue1667546. ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Patches for Pypy under cygwin
The workaround I implement is based on full-hour differences between localtime and gmtime, and uses hardcoded GMT+/-# strings. It obviously will not work correctly for timezones that have half-hour offsets to GMT, but it seemed to do the trick for what I needed. - Original Message - From: Alexander Belopolsky alexander.belopol...@gmail.com To: Uwe F. Mayer uwe_f_ma...@yahoo.com Cc: pypy-dev@python.org pypy-dev@python.org Sent: Wednesday, June 13, 2012 5:22 PM Subject: Re: [pypy-dev] Patches for Pypy under cygwin On Mon, Apr 30, 2012 at 7:05 PM, Uwe F. Mayer uwe_f_ma...@yahoo.com wrote: In particular the work-around for the tm structure to get by without the tm_gmtoff and tm_zone fields may be of interest, as it allows compilation of the rctime module on systems that don't have it, such as some versions of SunOS. Interesting and timely. I am about to add limited tm_gmtoff/tm_zone support to CPython. See http://bugs.python.org/issue1667546. ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Patches for Pypy under cygwin
I have made a final patch (as of now) and linked it from:
http://www.tux.org/~mayer/cygwin/pypy/index.html
See the bottom third of the page, the patch against the trunk of the mercurial
repository is
http://www.tux.org/~mayer/cygwin/pypy/pypy-pypy-ac392fb76904.patch
With this pypy builds under Cygwin, and os.fork() works.
It would be good if this could get merged into the trunk.
--Uwe
- Original Message -
From: Uwe F. Mayer uwe_f_ma...@yahoo.com
To: Armin Rigo ar...@tunes.org; Amaury Forgeot d'Arc amaur...@gmail.com
Cc: Matti Picus matti.pi...@gmail.com; pypy-dev@python.org
pypy-dev@python.org
Sent: Monday, June 11, 2012 10:27 AM
Subject: Re: [pypy-dev] Patches for Pypy under cygwin
- Original Message -
From: Armin Rigo ar...@tunes.org
Sent: Monday, June 11, 2012 4:59 AM
On Sun, Jun 10, 2012 at 11:12 AM, Amaury Forgeot d'Arc amaur...@gmail.com
wrote:
I think memory allocated for JIT must be marked with PROT_EXEC (it contains
machine code), so malloc() is not an option.
Exactly. On Linux it would work, because you can use mprotect() to
make even malloc()ed memory executable, but I doubt this non-Posix
extension works on cygwin. What might work instead would be using the
Windows API instead of the POSIX one. In pypy/rlib/rmmap.py, look for
def alloc and def free: there are the two versions.
Cygwin implements the POSIX API on top of the Windows API, and some things are
tricky to implement, and memory handling and permissions tend to be part of
those. To see if it may just work, I modified rmmap.pyto use malloc() and
free(), and indeed pypy seems to work just fine. Following up on this email I
also added an mprotect() call, which does exist under Cygwin. However, POSIX
says that the behavior of mprotect() is unspecified if it is applied to a
region of memory that was not obtained via mmap(2), and indeed this fails.
Here's the patch that works, using malloc() and free(), and not using
mprotect().
*** pypy/pypy/rlib/rmmap.pyMon Jun 11 10:18:55 2012
--- pypy/pypy/rlib/rmmap.pyMon Jun 11 10:16:12 2012
***
*** 14,19
--- 14,20
_MS_WINDOWS = os.name == nt
_LINUX = linux in sys.platform
_64BIT = 64bit in platform.architecture()[0]
+ _CYGWIN = cygwin == sys.platform
class RValueError(Exception):
def __init__(self, message):
***
*** 115,120
--- 116,126
PTR = rffi.CCHARP
+ if _CYGWIN:
+ c_malloc, _ = external('malloc', [size_t], PTR)
+ #c_mprotect, _ = external('mprotect', [PTR, size_t, rffi.INT], rffi.INT)
+ c_free, _ = external('free', [PTR], lltype.Void)
+
c_memmove, _ = external('memmove', [PTR, PTR, size_t], lltype.Void)
if _POSIX:
***
*** 692,714
so the memory has the executable bit set and gets allocated
internally in case of a sandboxed process.
! flags = MAP_PRIVATE | MAP_ANONYMOUS
! prot = PROT_EXEC | PROT_READ | PROT_WRITE
! hintp = rffi.cast(PTR, hint.pos)
! res = c_mmap_safe(hintp, map_size, prot, flags, -1, 0)
! if res == rffi.cast(PTR, -1):
! # some systems (some versions of OS/X?) complain if they
! # are passed a non-zero address. Try again.
! hintp = rffi.cast(PTR, 0)
! res = c_mmap_safe(hintp, map_size, prot, flags, -1, 0)
! if res == rffi.cast(PTR, -1):
raise MemoryError
else:
! hint.pos += map_size
return res
alloc._annenforceargs_ = (int,)
! free = c_munmap_safe
elif _MS_WINDOWS:
def mmap(fileno, length, tagname=, access=_ACCESS_DEFAULT, offset=0):
--- 698,739
so the memory has the executable bit set and gets allocated
internally in case of a sandboxed process.
! if _CYGWIN:
! # XXX: JIT memory should be using mmap MAP_PRIVATE with
! # PROT_EXEC but Cygwin's fork() fails
! res = c_malloc(map_size)
! if res == rffi.cast(PTR, 0):
raise MemoryError
+ # XXX POSIX says that the behavior of mprotect() is unspecified
+ # if it is applied to a region of memory that was not obtained
+ # via mmap(2), and indeed it does not work here.
+ # prot = PROT_EXEC | PROT_READ | PROT_WRITE
+ # res2 = c_mprotect(res, map_size, prot)
+ # if res2 == -1:
+ # errno = rposix.get_errno()
+ # raise OSError(errno, os.strerror(errno))
else:
! flags = MAP_PRIVATE | MAP_ANONYMOUS
! prot = PROT_EXEC | PROT_READ | PROT_WRITE
! hintp = rffi.cast(PTR, hint.pos)
! res = c_mmap_safe(hintp, map_size, prot, flags, -1, 0)
! if res == rffi.cast(PTR, -1):
! # some systems (some versions of OS/X?) complain if they
! # are passed a non-zero address. Try again
Re: [pypy-dev] Patches for Pypy under cygwin
Hi, On Sun, Jun 10, 2012 at 11:12 AM, Amaury Forgeot d'Arc amaur...@gmail.com wrote: I think memory allocated for JIT must be marked with PROT_EXEC (it contains machine code), so malloc() is not an option. Exactly. On Linux it would work, because you can use mprotect() to make even malloc()ed memory executable, but I doubt this non-Posix extension works on cygwin. What might work instead would be using the Windows API instead of the POSIX one. In pypy/rlib/rmmap.py, look for def alloc and def free: there are the two versions. A bientôt, Armin. ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Patches for Pypy under cygwin
2012/6/10 Uwe F. Mayer uwe_f_ma...@yahoo.com How hard would it be for Pypy to use malloc() instead of mmap() for Cygwin? I think memory allocated for JIT must be marked with PROT_EXEC (it contains machine code), so malloc() is not an option. -- Amaury Forgeot d'Arc ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Patches for Pypy under cygwin
Armin, thanks for merging in the patch. I have now cloned the latest version, and tried to compile under Cygwin. I ran into a couple more issues, attached is a follow-up patch. In parallel I had a longer conversation with Matti about potentially setting up a build slave for Pypy under Cygwin. Clearly some kind of testing is needed. Currently the Cygwin Pypy Python standalone version build with --opt=jit fails on os.fork() calls. The Pypy Python version build with --opt=2 handles os.fork() apparently just fine, on the other hand it cannot be used to build Pypy and simply exits without error message after platcheck_28. So the question is, do you have a regular build for Cygwin? I have not fully checked into whether I could set up a build slave, but it's clear already that because of the long time it takes to run a build (6.5 hours), it would have to be a weekly, not nightly, setup. I am willing to pursue that if it would help the project. --Uwe From: Armin Rigo ar...@tunes.org To: Uwe F. Mayer uwe_f_ma...@yahoo.com Cc: pypy-dev@python.org pypy-dev@python.org Sent: Sunday, June 3, 2012 2:57 AM Subject: Re: [pypy-dev] Patches for Pypy under cygwin Hi Uwe, On Tue, May 1, 2012 at 1:05 AM, Uwe F. Mayer uwe_f_ma...@yahoo.com wrote: I edited / hacked the sources and managed to compile PyPy under Cygwin based on Python 2.6.7 and gcc-4.5.3. If you find this of interest, the patches are available at http://www.tux.org/~mayer/cygwin/pypy/. Thanks! I finally took the time to review and merge the patch. I didn't include any change to lib-python: they are changes that look like they should also be done to CPython..? I may be mistaken, but it looks like fixes for tests that would also fail on CPython+cygwin. If that's the case, then it should be reported and fixed in CPython. A bientôt, Armin. pypy.patch.no-python Description: Binary data ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Patches for Pypy under cygwin
2012/6/7 Uwe F. Mayer uwe_f_ma...@yahoo.com Currently the Cygwin Pypy Python standalone version build with --opt=jit fails on os.fork() calls. fork() on Windows... I'm surprised it works at all! You should probably read this page: http://cygwin.com/cygwin-ug-net/highlights.html#ov-hi-process specially the DLL base address collisions, did you run rebaseall? Is pypy linked with some non-cygwin load-time DLLs? -- Amaury Forgeot d'Arc ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Patches for Pypy under cygwin
The pypy-c executable is not a dll, and hence does not need to be rebased if I understand it correctly. However, just to follow up on this hunch, I rebased it by first running rebaseall -v on the complete Cygwin installation (which does not touch pypy-c even if put into /usr/bin/) and then manually rebasing pypy-c to the next available base address. Unfortunately this does not help. As to the question about runtime libraries, pypy links against the following libraries: $ cygcheck ./pypy-c D:\pypy\pypy-work-no-python-patch\pypy\pypy\translator\goal\pypy-c C:\cygwin\bin\cygbz2-1.dll C:\cygwin\bin\cygwin1.dll C:\windows\system32\KERNEL32.dll C:\windows\system32\ntdll.dll C:\cygwin\bin\cyggcc_s-1.dll C:\cygwin\bin\cygcrypt-0.dll C:\cygwin\bin\cygcrypto-1.0.0.dll C:\cygwin\bin\cygz.dll C:\cygwin\bin\cygncurses-10.dll C:\cygwin\bin\cygexpat-1.dll C:\cygwin\bin\cygintl-8.dll C:\cygwin\bin\cygiconv-2.dll C:\cygwin\bin\cygssl-1.0.0.dll C:\cygwin\bin\cygffi-4.dll Just for good measure, here's a typical error message from an interactive pypy session: $ pypy-c Python 2.7.2 (b0005ab7c34f+, Jun 06 2012, 17:33:28) [PyPy 1.9.1-dev0 with GCC 4.5.3] on cygwin Type help, copyright, credits or license for more information. And now for something completely different: ``'that's definitely a case of uh''' import os os.fork() 3 [main] pypy-c 11788 fixup_mmaps_after_fork: ReadProcessMemory failed for MAP_PRIVATE address 0xB001, Win32 error 998 153 [main] pypy-c 11788 D:\pypy\pypy-work-no-python-patch\pypy\pypy\translator\goal\pypy-c: *** fatal error in forked process - recreate_mmaps_after_fork_failed 499 [main] pypy-c 11788 open_stackdumpfile: Dumping stack trace to pypy-c.stackdump 2 [main] pypy-c 2572 fork: child -1 - forked process 11788 died unexpectedly, retry 0, exit code 256, errno 11 Traceback (most recent call last): File console, line 1, in module OSError: [Errno 11] Resource temporarily unavailable I noticed that the address 0xB001 in the error message above seems to be the same regardless of rebasing pypy-c or not, and I am seeing this address on two different Windows 2003 64bit servers. For what it's worth, the expected output, from a python session, would be: $ python Python 2.6.7 (r267:88850, Feb 2 2012, 23:50:20) [GCC 4.5.3] on cygwin Type help, copyright, credits or license for more information. import os os.fork() 5552 0 Finally, today unexpectedly the os.fork() call works on a Windows XP Professional 32bit box, where it previously failed (that's the Cygwin pypy 1.8.1 version I had posted on my web page). --Uwe From: Amaury Forgeot d'Arc amaur...@gmail.com To: Uwe F. Mayer uwe_f_ma...@yahoo.com Cc: Armin Rigo ar...@tunes.org; Matti Picus matti.pi...@gmail.com; pypy-dev@python.org pypy-dev@python.org Sent: Thursday, June 7, 2012 10:46 AM Subject: Re: [pypy-dev] Patches for Pypy under cygwin 2012/6/7 Uwe F. Mayer uwe_f_ma...@yahoo.com Currently the Cygwin Pypy Python standalone version build with --opt=jit fails on os.fork() calls. fork() on Windows... I'm surprised it works at all! You should probably read this page: http://cygwin.com/cygwin-ug-net/highlights.html#ov-hi-process specially the DLL base address collisions, did you run rebaseall? Is pypy linked with some non-cygwin load-time DLLs? -- Amaury Forgeot d'Arc___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Patches for Pypy under cygwin
As requested, I ran the code with strace. More precisely, I ran from the pypy/pypy/translator/goal/ directory the command: strace sh -c 'pypy-c /d/pypy/test_os_fork.py' I ran this twice, once with a version of pypy built with --opt=2, and once with with a version of pypy built with --opt=jit. The --opt=2 version succeeds, and the --opt=jit version fails, as reported earlier. Attached you'll find the test_os_fork.py script, and the two log files cut down to the relevant part. The log is a couple of MB, I cut it down to the relevant section. I don't know the answers the other two questions. The third question may touch the core of the issue if the JIT marks the (parent) memory as exclusive access before the fork() call, which would be consistent with the observed error. From what I can tell the fork() call never gets off the ground, having no read access to duplicate the process image. --Uwe From: Amaury Forgeot d'Arc amaur...@gmail.com To: Uwe F. Mayer uwe_f_ma...@yahoo.com Cc: Armin Rigo ar...@tunes.org; Matti Picus matti.pi...@gmail.com; pypy-dev@python.org pypy-dev@python.org Sent: Thursday, June 7, 2012 2:24 PM Subject: Re: [pypy-dev] Patches for Pypy under cygwin 2012/6/7 Uwe F. Mayer uwe_f_ma...@yahoo.com 3 [main] pypy-c 11788 fixup_mmaps_after_fork: ReadProcessMemory failed for MAP_PRIVATE address 0xB001, Win32 error 998 Aha, looks interesting. Searching for this error message, I found that cygwin unblocks the parent process before the child calls fixup_mmaps_after_fork: http://www.mail-archive.com/cygwin@cygwin.com/msg05547.html Could you try again with strace? Does the parent really unmap memory? Or is it because of the special access rights that the JIT uses to mark the memory as executable? -- Amaury Forgeot d'Arcimport sys, os pid = os.fork() if pid == 0: print I'm the child, my PID is, os.getpid() sys.exit() else: print I'm the parent, my PID is, os.getpid(), the child PID is, pid sys.exit() test_os_fork_opt=2.log Description: Binary data test_os_fork_opt=jit.log Description: Binary data ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Patches for Pypy under cygwin
Hi Uwe, On Thu, Jun 7, 2012 at 9:21 PM, Uwe F. Mayer uwe_f_ma...@yahoo.com wrote: 3 [main] pypy-c 11788 fixup_mmaps_after_fork: ReadProcessMemory failed for MAP_PRIVATE address 0xB001, Win32 error 998 Yes, that's one of the mmap addresses used by the JIT, if available. More to the point, googling for fork cygwin mmap shows that there are open bugs with cygwin's fork() in the presence of MAP_PRIVATE mmaps. That's obviously why it fails. I don't really know what we can reasonably do about that, short of asking people to use some specific versions of cygwin where it seems to work. I can imagine what we could unreasonably do as work-around, but that requires a lot of work and gross hacks that don't seem appropriate to put in the pypy source just to support cygwin... I would recommend that you try to see it fixed on the cygwin side. A bientôt, Armin. ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
[pypy-dev] Patches for Pypy under cygwin
I edited / hacked the sources and managed to compile PyPy under Cygwin based on Python 2.6.7 and gcc-4.5.3. If you find this of interest, the patches are available at http://www.tux.org/~mayer/cygwin/pypy/. In particular the work-around for the tm structure to get by without the tm_gmtoff and tm_zone fields may be of interest, as it allows compilation of the rctime module on systems that don't have it, such as some versions of SunOS. --Uwe___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
