Re: Failure during build of Python 3.8 via cygport
On 2020-12-13 07:34, Marco Atzeri via Cygwin-apps wrote: On 13.12.2020 09:52, Mark Geisert wrote: Mark Geisert wrote: This seems to be a problem setting up a platform-specific build directory. The sysconfig.py script wants to use "lib." + platform + pythonversion but the platform string somehow gets corrupted into non-utf8 bytes. For instance, building Python 3.8 comes up with: lib.cygwin-\365\377\377o-\377o-3.8 as the directory name. Broken, but could work. The build failure happens because the script tries to write this directory name into a file but it's not a valid utf8 string. The directory name should have been: lib.cygwin-3.2.0-x86_64-3.8 And the corruption is due to something about a recent change to the operation of Cygwin's uname() function. The change was introduced in Cygwin API version 335; I'm running 340 on my test machine. This being a fairly recent change might possibly explain why nobody else has run into this issue yet. That was nearly two years ago. Basically, os.uname within Python is calling Cygwin's uname() passing the address of a buffer declared to be 'struct utsname'. The structure layout changed in API 335. What I've hit is a mismatch between what Python expects and Cygwin delivers. I'll move this discussion over to the developers list tomorrow. uname(2): NOTES ... The length of the fields in the struct varies. Some operating systems or libraries use a hardcoded 9 or 33 or 65 or 257. Other systems use *SYS_NMLN* or *_SYS_NMLN* or *UTSLEN* or *_UTSNAME_LENGTH*. Clearly, it is a bad idea to use any of these constants; just use *sizeof*(...). Often 257 is chosen in order to have room for an internet hostname. ... C library/kernel differences Over time, increases in the size of the /utsname/ structure have led to three successive versions of *uname*(): /sys_olduname/()(slot /__NR_oldolduname/), /sys_uname/() (slot /__NR_olduname/), and /sys_newuname/()(slot /__NR_uname/). The first one used length 9 for all fields; the second used 65; the third also uses 65 but adds the domainname field. The glibc *uname*() wrapper function hides these details from applications, invoking the most recent version of the system call provided by the kernel. thanks for the analysis let me know if you think we should correct the python build in any way -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised. [Data in binary units and prefixes, physical quantities in SI.]
Re: Failure during build of Python 3.8 via cygport
On 13.12.2020 09:52, Mark Geisert wrote: Mark Geisert wrote: This seems to be a problem setting up a platform-specific build directory. The sysconfig.py script wants to use "lib." + platform + pythonversion but the platform string somehow gets corrupted into non-utf8 bytes. For instance, building Python 3.8 comes up with: lib.cygwin-\365\377\377o-\377o-3.8 as the directory name. Broken, but could work. The build failure happens because the script tries to write this directory name into a file but it's not a valid utf8 string. The directory name should have been: lib.cygwin-3.2.0-x86_64-3.8 And the corruption is due to something about a recent change to the operation of Cygwin's uname() function. The change was introduced in Cygwin API version 335; I'm running 340 on my test machine. This being a fairly recent change might possibly explain why nobody else has run into this issue yet. Basically, os.uname within Python is calling Cygwin's uname() passing the address of a buffer declared to be 'struct utsname'. The structure layout changed in API 335. What I've hit is a mismatch between what Python expects and Cygwin delivers. I'll move this discussion over to the developers list tomorrow. ..mark thanks for the analysis let me know if you think we should correct the python build in any way Regards Marco
Re: Failure during build of Python 3.8 via cygport
Mark Geisert wrote: This seems to be a problem setting up a platform-specific build directory. The sysconfig.py script wants to use "lib." + platform + pythonversion but the platform string somehow gets corrupted into non-utf8 bytes. For instance, building Python 3.8 comes up with: lib.cygwin-\365\377\377o-\377o-3.8 as the directory name. Broken, but could work. The build failure happens because the script tries to write this directory name into a file but it's not a valid utf8 string. The directory name should have been: lib.cygwin-3.2.0-x86_64-3.8 And the corruption is due to something about a recent change to the operation of Cygwin's uname() function. The change was introduced in Cygwin API version 335; I'm running 340 on my test machine. This being a fairly recent change might possibly explain why nobody else has run into this issue yet. Basically, os.uname within Python is calling Cygwin's uname() passing the address of a buffer declared to be 'struct utsname'. The structure layout changed in API 335. What I've hit is a mismatch between what Python expects and Cygwin delivers. I'll move this discussion over to the developers list tomorrow. ..mark
Re: Failure during build of Python 3.8 via cygport
[replying to myself again...] A similar problem happens when building 3.6 and 3.7 too. Details at end. On Wed, 9 Dec 2020, Mark Geisert wrote: Hi Marco, I was building Python locally so I can later submit a patch against it. It appears the local python.exe was built successfully, but a later step failed with: ./python.exe -E -S -m sysconfig --generate-posix-vars ;\ if test $? -ne 0 ; then \ echo "generate-posix-vars failed" ; \ rm -f ./pybuilddir.txt ; \ exit 1 ; \ fi Traceback (most recent call last): File "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/runpy.py", line 87, in _run_code exec(code, run_globals) File "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/sysconfig.py", line 711, in _main() File "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/sysconfig.py", line 699, in _main _generate_posix_vars() File "/usr/src/python38-3.8.3-1.src/python38-3.8.3-1.x86_64/src/Python-3.8.3/Lib/sysconfig.py", line 416, in _generate_posix_vars f.write(pybuilddir) UnicodeEncodeError: 'utf-8' codec can't encode characters in position 17-19: surrogates not allowed generate-posix-vars failed make: *** [Makefile:592: pybuilddir.txt] Error 1 *** ERROR: make failed I've seen UnicodeEncodeError before and searched and found how to fix it.. but hitting the issue while building Python itself seems more fraught. Is this a known issue with known fix? This seems to be a problem setting up a platform-specific build directory. The sysconfig.py script wants to use "lib." + platform + pythonversion but the platform string somehow gets corrupted into non-utf8 bytes. For instance, building Python 3.8 comes up with: lib.cygwin-\365\377\377o-\377o-3.8 as the directory name. Broken, but could work. The build failure happens because the script tries to write this directory name into a file but it's not a valid utf8 string. The directory name should have been: lib.cygwin-3.2.0-x86_64-3.8 I'm trying to debug further, learning Python as I go. Whee ..mark
