Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
I'm following up on this thread without checking if there were other
following negating a need to respond... If so, ignore as needed.
+1 from me. Always build on windows into an architecture specific
PCBuild/XXX directory. A bonus if the directory name matches the return
value of platform.machine() but I don't care too much about that part.
(that'd mean 'i686' and 'x86_64' and who knows what for Itanic users)
On Wed, Jan 30, 2008 at 11:25 PM, Mark Hammond [EMAIL PROTECTED]
wrote:
I'm wondering if anyone has any comment on this issue? Its currently
impossible to use distutils as documented to build x64 extensions, either
natively or cross-compiled using the trunk and while I'm keen to help fix
this, I'd like agreement on the direction.
Can I assume silence means general agreement on the compromise proposal I
outlined?
Thanks,
Mark
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On
Behalf Of Mark Hammond
Sent: Tuesday, 22 January 2008 9:06 PM
To: 'Martin v. Löwis'
Cc: [EMAIL PROTECTED]; python-dev@python.org
Subject: Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Hi Martin,
Way back on Monday, 21 May 2007, you wrote:
This is an issue to be discussed for Python 2.6. I'm personally
hesitant to have the official build infrastructure deviate
from the layout that has been in-use for so many years, as a lot
of things depend on it.
I don't find the need to have separate object directories convincing:
For building the Win32/Win64 binaries, I have separate checkouts
*anyway*, since all the add-on libraries would have to support
multi-arch builds, but I think they don't.
...
* Re the x64 directory used by the PCBuild8 process. IMO, it makes
sense to generate x64 binaries to their own directory - my
expectation
is that cross-compiling between platforms is a reasonable use-case,
and we should support multiple achitectures for the same compiler
version.
See above; I disagree.
[For additional context; the question was regarding where the output
binaries were created for each platform, and specifically if x64 build
should use something like 'PCBuild/x64']
I'm bringing this topic up again as I noticed the AMD64 builds for
Python
2.6 create their output in the PCBuild/x64 directory, not directly into
the
PCBuild directory. When I raised this with Christian, he also
expressed a
desire to be able to cross-compile in the same directory structure and
was
unaware of this discussion (but I made him aware of it :)
You made excellent points about how tools currently recognize the
PCBuild
directory, and we can't break them, and I agree. I'd like to propose a
compromise that might be able to keep everyone happy.
The main problem I see with all platforms using 'PCBuild' is that in a
cross-compilation scenario, it is impossible for the distutils to
determine
the location of the correct Python libraries to link with (stuff in its
own
PYTHONHOME is no good; it's for the wrong platform). Obviously, we can
change the distutils so that the location of the correct libraries can
be
specified, but that implies all other build systems that currently
assume
PCBuild must also change to work in a cross-compilation scenario.
While
those external tools *will* work today in a native build on x64,
eventually
they may need to be upgraded to deal with cross-compilation - and this
means
that all such tools will also be unable to automatically determine the
location of the libraries. They will all need to take the library dir
as a
user-specified option, which seems a pain (eg, buildbots would seem to
need
site-specific configuration information to make this work). If
eventually
all such tools are likely to upgrade, it would be ideal if we can offer
them
a way to upgrade so the correct libraries can be determined
automatically,
as they can now for native builds.
My compromise proposal is this: Each platform builds in its own
directory
(ie, PCBuild/x86_64, PCBuild/x86). Every single build configuration
has a
post-build step that copies the result of the build to the PCBuild
directory. We then add an option to the distutils so that a cross-
compile
platform can be specified and when it is, to use 'PCBuild/{platform}
instead of PCBuild, and we encourage other tools to use the same
directory
should they want to support configure-less cross-compilation (if
distutils
on the trunk is not still trying to maintain b/w compat, it should just
*always* look in PCBuild/{platform}, and I'd suggest py3k not bother
with
PCBuild at all; build systems will be touched to upgrade to py3k, so
that
change isn't painful. But I digress :)
The main advantage of the compromise is that it allows for the status
quo to
remain in place - just continue to use separate source trees for each
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Hi Martin,
Way back on Monday, 21 May 2007, you wrote:
This is an issue to be discussed for Python 2.6. I'm personally
hesitant to have the official build infrastructure deviate
from the layout that has been in-use for so many years, as a lot
of things depend on it.
I don't find the need to have separate object directories convincing:
For building the Win32/Win64 binaries, I have separate checkouts
*anyway*, since all the add-on libraries would have to support
multi-arch builds, but I think they don't.
...
* Re the x64 directory used by the PCBuild8 process. IMO, it makes
sense to generate x64 binaries to their own directory - my expectation
is that cross-compiling between platforms is a reasonable use-case,
and we should support multiple achitectures for the same compiler
version.
See above; I disagree.
[For additional context; the question was regarding where the output
binaries were created for each platform, and specifically if x64 build
should use something like 'PCBuild/x64']
I'm bringing this topic up again as I noticed the AMD64 builds for Python
2.6 create their output in the PCBuild/x64 directory, not directly into the
PCBuild directory. When I raised this with Christian, he also expressed a
desire to be able to cross-compile in the same directory structure and was
unaware of this discussion (but I made him aware of it :)
You made excellent points about how tools currently recognize the PCBuild
directory, and we can't break them, and I agree. I'd like to propose a
compromise that might be able to keep everyone happy.
The main problem I see with all platforms using 'PCBuild' is that in a
cross-compilation scenario, it is impossible for the distutils to determine
the location of the correct Python libraries to link with (stuff in its own
PYTHONHOME is no good; it's for the wrong platform). Obviously, we can
change the distutils so that the location of the correct libraries can be
specified, but that implies all other build systems that currently assume
PCBuild must also change to work in a cross-compilation scenario. While
those external tools *will* work today in a native build on x64, eventually
they may need to be upgraded to deal with cross-compilation - and this means
that all such tools will also be unable to automatically determine the
location of the libraries. They will all need to take the library dir as a
user-specified option, which seems a pain (eg, buildbots would seem to need
site-specific configuration information to make this work). If eventually
all such tools are likely to upgrade, it would be ideal if we can offer them
a way to upgrade so the correct libraries can be determined automatically,
as they can now for native builds.
My compromise proposal is this: Each platform builds in its own directory
(ie, PCBuild/x86_64, PCBuild/x86). Every single build configuration has a
post-build step that copies the result of the build to the PCBuild
directory. We then add an option to the distutils so that a cross-compile
platform can be specified and when it is, to use 'PCBuild/{platform}
instead of PCBuild, and we encourage other tools to use the same directory
should they want to support configure-less cross-compilation (if distutils
on the trunk is not still trying to maintain b/w compat, it should just
*always* look in PCBuild/{platform}, and I'd suggest py3k not bother with
PCBuild at all; build systems will be touched to upgrade to py3k, so that
change isn't painful. But I digress :)
The main advantage of the compromise is that it allows for the status quo to
remain in place - just continue to use separate source trees for each
platform, and only ever build a single platform in each tree, and tools are
free to have ways of specifying which directory should be used. The
official build process need not change. However, it also supports a way to
do cross-compilation in a way that build tools can *automatically* locate
the correct platform's libraries, and this will be particularly useful for
extension authors.
Would something like that be acceptable?
Thanks,
Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
-Original Message- From: Martin v. Löwis [mailto:[EMAIL PROTECTED] Just imagine the a school teacher who in good faith wants to introduce his pupils to the wonderful programming language of Python, but when he installs it, all kinds of scary looking warnings drive him off. Vista is, like it or not, going to be very prevalent. If we want python to be easily accessible to the masses, we mustn't take an elitist attitude or else risk scaring people off. I'm completely in favor of fixing actual bugs. However, I'm not aware of any (related to Vista). That it is not logo compliant is *not* a bug. Python hasn't been logo compliant for more than a decade now (the install to Program Files is not a new requirement, but existed since Win95). I'm not saying that it is a bug, but since this is python-dev, I am discussing it as a desirable feature. One feature that is easily addable and will certainly make installing python on vista nicer, is to add authenticode signing to the install. Currently the user is faced with a very nasty and off-putting message about an unidentified program requesting access to his computer. See http://msdn2.microsoft.com/en-us/library/bb172338.aspx . I think the PSF should be able to obtain a certificate from MS. cheers, Kristján ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
At 1:14 PM + 5/29/07, Kristján Valur Jónsson wrote: -Original Message- Microsoft's command line cannot cope with two pathnames that must be quoted, so if the command path itself must be quoted, then no argument to the command can be quoted. There are tricky hacks that can work around this mind-boggling stupidity, but life is simpler if Python itself doesn't use up the one quoted pathname. I don't know if Microsoft has had the good sense to fix this in Vista (which I probably will never use, since an alternative exists), but they didn't in XP. Do you have any references for this claim? In my command line on XP sp2, this works just fine: C:\Program Files\Microsoft Visual Studio 8\VCc:\Program Files\TextPad 4\TextPad.exe c:\tmp\f a.txt c:\tmp\f b.txt Both the program, and the two file names are quoted and textpad.exe opens them both. I pounded my head against this issue when working on a .bat file a few years back, until I read the help for cmd and saw the quote logic (and switched to VBScript). It's still there, in help cmd. I had once found references to the same issue for the run command in Microsoft's online help. Perhaps it is fixed in SP2. If so, just change it and don't worry about users with earlier versions of Windows. -- TonyN.:' mailto:[EMAIL PROTECTED] ' http://www.georgeanelson.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
One feature that is easily addable and will certainly make installing python on vista nicer, is to add authenticode signing to the install. This I question very much. I experimented with authenticode before 2.4, and found it an unacceptable experience. When the MSI file starts running, installer needs to verify the signature, for which it needs to compute a hash of the entire file. For the Python MSI, this takes many seconds on a slower Pentium 4 machine. During that time, there is no visual feedback, so users are uncertain whether they have actually invoked the MSI file at all. Currently the user is faced with a very nasty and off-putting message about an unidentified program requesting access to his computer. Certainly. However, telling them that they have to wait just so that Windows finds out what they know already (that this is the MSI file from the Python Software Foundation, or from Martin v. Löwis) is even more nasty. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
It doesn't need to, and reluctance is not wrt. to the proposed new layout, but wrt. changing the current one. Tons of infrastructure depends on the files having exactly the names that they have now, and being located in exactly the locations where they are currently located. Any change to that, whatever minor, will cause problems to some people. Just to be absolutely clear: You are talking about the build environment, right? Because I am not proposing to change any layout of the installed Python (wherever that may be :) Correct. I am baffled about why the build environment's layout matters, but once an .msi install can place the binaries in any old place it wants. The build structure doesn't have to reflect the final installed structure at all. No. But still, people like to be able to run Python out of a source check-out. This has been supported for a long time, and more and more stuff was added to support it. For examples within Python itself, see the support in distutils, getpathp.c, PCbuild/rt.bat, and Tools/buildbot/*.bat. Reportedly (by Mark), building Mozilla (the web browser) also knows about PCbuild. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Development tools used on windows already have to cope with this. Spaces are not going away, so why not bite the bullet and deal with them? Moving forward sometimes means crossing rivers. But in a safe path, step by step. People continue to report problems with spaces in file names, even though many of them have been solved also. As for Vista issues, I'll gather more data before making any more claims, but I think that it is important that we play by the rules here. I completely disagree. It is important that it works, not that we play by the rules. Just imagine the a school teacher who in good faith wants to introduce his pupils to the wonderful programming language of Python, but when he installs it, all kinds of scary looking warnings drive him off. Vista is, like it or not, going to be very prevalent. If we want python to be easily accessible to the masses, we mustn't take an elitist attitude or else risk scaring people off. I'm completely in favor of fixing actual bugs. However, I'm not aware of any (related to Vista). That it is not logo compliant is *not* a bug. Python hasn't been logo compliant for more than a decade now (the install to Program Files is not a new requirement, but existed since Win95). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Supporting both kinds (country and western) on the same machine might be helpful to people for this very reason. A lot of legacy modules are only avaible in 32 bit mode. But people may want to do contemporary development using the new 64 bit mode. Of course, people who really want that can install both versions simultaneously today. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
-Original Message- From: Martin v. Löwis [mailto:[EMAIL PROTECTED] It doesn't need to, and reluctance is not wrt. to the proposed new layout, but wrt. changing the current one. Tons of infrastructure depends on the files having exactly the names that they have now, and being located in exactly the locations where they are currently located. Any change to that, whatever minor, will cause problems to some people. Just to be absolutely clear: You are talking about the build environment, right? Because I am not proposing to change any layout of the installed Python (wherever that may be :) I am baffled about why the build environment's layout matters, but once an .msi install can place the binaries in any old place it wants. The build structure doesn't have to reflect the final installed structure at all. Kristján ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
-Original Message- From: Alexey Borzenkov [mailto:[EMAIL PROTECTED] Sent: Wednesday, May 23, 2007 20:36 To: Kristján Valur Jónsson Cc: Martin v. Löwis; Mark Hammond; [EMAIL PROTECTED]; python- [EMAIL PROTECTED] Subject: Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows On 5/23/07, Kristján Valur Jónsson [EMAIL PROTECTED] wrote: Install in the ProgramFiles folder. Only over my dead body. *This* is silly. Bill doesn't think so. And he gets to decide. I mean we do want to play nice, don't we? Nothing installs itself in the root anymore, not since windows 3.1 Maybe installing in the root is not good, but installing to Program Files is just asking for trouble. All sorts of development tools might suddenly break because of that space in the middle of the path and requirement to use quotes around it. I thus usually install things to drive:\Programs. I'm not sure if any packages/programs will break because of that space, but what if some will? Development tools used on windows already have to cope with this. Spaces are not going away, so why not bite the bullet and deal with them? Moving forward sometimes means crossing rivers. As for Vista issues, I'll gather more data before making any more claims, but I think that it is important that we play by the rules here. Just imagine the a school teacher who in good faith wants to introduce his pupils to the wonderful programming language of Python, but when he installs it, all kinds of scary looking warnings drive him off. Vista is, like it or not, going to be very prevalent. If we want python to be easily accessible to the masses, we mustn't take an elitist attitude or else risk scaring people off. Finally, to add a (mis)quote from Mr. Gorbachev: Wer zu spät kommt, den bestraft das Leben Kristján ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
-Original Message-
I personally think that if hostile users can replace DLLs on your
system, you have bigger problems than SxS installation of
pythhonxy.dll,
but perhaps that's just me.
An end user application on an end-user's machine is always voulnerable
to reverse engineering. But it helps to make it more difficult.
The old-style dll loader is a particular vulnerability which makes
it easy to patch into an application at load time.
What about the registry?
I don't know about the registry, what is it used for?
For two things, with different importance to different users:
1. File extensions are registered there, e.g. .py and .pyc.
With two binaries installed, the will stomp over each other's
file associations; only one of them can win.
Sure. No argument about this. But as with the explorer and other apps,
it is perfectly possible to manually start one or the other, autoclicking
on .py files isn't the only option.
2. Python installs keys under
HL{LM|CU}\Software\Python\PythonCore\version, namely
InstallPath
InstallGroup
PythonPath
Documentation
Modules
Funnily enough, Bill has thought of this too. See
http://support.microsoft.com/kb/305097/
for info. Co-existence of 32 and 64 bit apps is supported.
The two versions of MSIE actually *are* a big problem, that's
why MS only runs the 32-bit IE, even on Win64 (otherwise, ActiveX
controls downloaded from the net wouldn't work).
Also, while they are both shipped, you can't the two versions of
explorer.exe simultaneously (without trickery), so its far from
simple.
The two versions aren't the problem, it is the backward support for
32 bit active thingies that are, as you point out. There is confusion here:
internet explorer shipped in both versions. The 32 bit version war default
for the above reason. But explorer.exe (which I was talking about) also
had two versions. The 64 bit version ran by default.
You may recall that before tortoise shipped a 64 bit version, one had to
kill explorer.exe and restart it (explorer32.exe IIRC) to get tortoise
to work.
Supporting both kinds (country and western) on the same machine might be helpful
to people for this very reason. A lot of legacy modules are only avaible
in 32 bit mode. But people may want to do contemporary development using the
new 64 bit mode.
Cheers,
Kristján
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
At 12:20 PM + 5/26/07, Kristján Valur Jónsson wrote: -Original Message- From: Alexey Borzenkov [mailto:[EMAIL PROTECTED] Sent: Wednesday, May 23, 2007 20:36 To: Kristján Valur Jónsson Cc: Martin v. Löwis; Mark Hammond; [EMAIL PROTECTED]; python- [EMAIL PROTECTED] Subject: Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows On 5/23/07, Kristján Valur Jónsson [EMAIL PROTECTED] wrote: Install in the ProgramFiles folder. Only over my dead body. *This* is silly. Bill doesn't think so. And he gets to decide. I mean we do want to play nice, don't we? Nothing installs itself in the root anymore, not since windows 3.1 Maybe installing in the root is not good, but installing to Program Files is just asking for trouble. All sorts of development tools might suddenly break because of that space in the middle of the path and requirement to use quotes around it. I thus usually install things to drive:\Programs. I'm not sure if any packages/programs will break because of that space, but what if some will? Development tools used on windows already have to cope with this. Spaces are not going away, so why not bite the bullet and deal with them? Moving forward sometimes means crossing rivers. ... Microsoft's command line cannot cope with two pathnames that must be quoted, so if the command path itself must be quoted, then no argument to the command can be quoted. There are tricky hacks that can work around this mind-boggling stupidity, but life is simpler if Python itself doesn't use up the one quoted pathname. I don't know if Microsoft has had the good sense to fix this in Vista (which I probably will never use, since an alternative exists), but they didn't in XP. -- TonyN.:' mailto:[EMAIL PROTECTED] ' http://www.georgeanelson.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
-Original Message- though, not cygwin (a la bsmedberg's new MozillaBuild stuff). I just wish there were an autoconf alternative that wasn't as painful as autoconf. I have a few attempts for my purposes that are written in Python (an obvious bootstrapping problem for building Python itself :). Only for the theorist. As you, we use build tools for our stackless branch written in python. There exist successfully built python versions back from the nineties, which can be considered external tools for all practical purposes. Building python with python is really nifty. Kristjan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
-Original Message- From: Martin v. Löwis [mailto:[EMAIL PROTECTED] That couldn't work for me. I try avoid building on a network drive, and with local drives, I just can't have a Windows build and a Linux build on the same checkout - they live on separate file systems, after all (Linux on ext3, Windows on NTFS, with multi-boot switching between them). Very well, leaving linux aside, I don't see why this: /win32mount/trunk/PCbuild/ /x64mount/trunk/PCbuild/ Is any different from /winmount/trunk/PCBuild/win32 /winmount/trunk/PCBuild/x64 I don't understand this extraordinary reluctance to add a single extra directory. The windows build process is different from any other build process, so even If all the other platforms live one directory higher, why must windows? I don't *use* them simultaneously, of course - I cannot work on two architectures simultaneously, anyway. I do so on a daily basis. I designed PCBuild8 to be easy for interactive work using VisualStudio, using property sheets and such to group common settings for easy editing. During the course of my work, I will edit a file (which is checked out from Perforce), compile debug for two platforms, test, and repeat. I say let's just admit that tools can compile for more than one target. Let's adapt to it and we will all be happier. You might be; I will be sad. It comes for a price, I well understand the benefits, I use it all the time, but the price still eludes me. how can a different name for the output folder for a different platform be such a big problem? And btw, there is no need to install the msvcr8.dll. We can distribute them as a private assembly. then they (and the manifest) exist in the same directory as python2x.dll. Yes, but then python2x.dll goes into system32, and so will msvcr8.dll, no? Yes, that is correct. Well, there is a CRT .exe redist if you want to deploy this into the SxS cache, it just has to be run as part of the install process. But that may or may not be problematic, I don't know. Not sure whether anything really is needed. Python works fine on Vista. If you are an administrator. A limited user will have problems installing it and then running it. Is there a bug report for that? I don't know. At any rate, I think any vista issues is a completely separate thing, something that needs to be handled as a whole, rather than responding to a particular problem reported in a bug report. 1) supplying python.dll as a Side By Side assembly What would that improve? Well, it should reduce dll-hell problems of applications that ship with python2x.dll. You ship with and link to your own and tested dll. We have some concerns here, for example, now that we are moving away from embedding python in our blue.dll and using python25.dll directly, that this exposes a vulnerability to the integrity of the software. Why should there be versioning problems with python25.dll? Are there any past issues with incompatibilities with any python2x.dll release? Someone could replace the python25.dll that we ship with their own patched version, thereby gaining backdoor access to the software. The way windows searches for old style dlls makes this easy. Using the SxS signed loading scheme, you can protect yourself up to a point from such attacks. Of course, this doesn't have to be a problem for vanilla python, we can do this on our own for the patched python25 that we employ, but it still might be something others could find useful. Install in the ProgramFiles folder. Only over my dead body. *This* is silly. Bill doesn't think so. And he gets to decide. I mean we do want to play nice, don't we? Nothing installs itself in the root anymore, not since windows 3.1 Just as C does. Ah, and this also means that we could install both 32 bit and 64 bit versions, another plus. What about the registry? I don't know about the registry, what is it used for? 64 bit windows already ships with dual versions of some apps, notably explorer.exe so that shouldn't be a big problem. Interesting. We are definitely interested in that. You see, Someone installs a game or accounting software using vista. He then runs as a limited user. Python insists on saving its .pyc files in the installation folder, but this is not something that is permitted on Vista. But that's not a problem, is it? Writing silently fails, i.e. it just won't save the pyc files. Happens all the time on Unix. It may not silently fail, depending on your user status. An admin might get a confirmation window, for example. Sure, and have they reported problems with Python on Vista (problems specific to Vista?) Certainly. We are working on them, of course. But, of course, they have not been reported. These are not python errors as such, but rather EVE errors. We ship the .py files precompiled and therefore avoid the aforementioned problems. But we have had to move all sorts of
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
[MarkH] I'm guessing vsextcomp doesn't use the Visual Studio 'ReleaseAMD64' configuration - would it be OK for me to check in changes to the PCBuild projects for this configuration? [Martin v. Löwis] Please don't. It exclusively relies on vsextcomp, and is only useful if you have that infrastructure installed. See for yourself: it uses the /USE_CL:MS_OPTERON command line switch, which isn't a Microsoft invention (but instead invented by Peter Tröger). Aside: it isn't my experience that vsextcomp is necessary to cross-compile for AMD64 and IA64. My cross-build process basically equates to: - Run the appropriate environment setup for the correct compiler. E.g., for the Platform SDK AMD64 compiler and with the current Platform SDK this is: C:\Program Files\Microsoft Platform SDK\SetEnv.Cmd /X64 /RETAIL - Run the solution file with devenv.com (IIRC, devenv.exe doesn't take command-line args) and be sure to pass ing /useenv to pick up the environment changes. (*) set DEVENV_COM=path/to/devenv.com %DEVENV_COM% PCbuild\pcbuild.sln /useenv /build ReleaseAMD64 (*) Note that for a cross-build the make_versioninfo and make_buildinfo projects need to be built natively first: C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\vcvars32.bat %DEVENV_COM% PCbuild\pcbuild.sln /useenv /build ReleaseAMD64 /project make_versioninfo %DEVENV_COM% PCbuild\pcbuild.sln /useenv /build ReleaseAMD64 /project make_buildinfo This is all VS7.1, though. I don't yet know if VS8 throws a spanner into the works. For VS6 I use msdev instead of devenv.com and PC\VC6\pcbuild.dsw instead of PCbuild\pcbuild.sln. I haven't looked into what vsextcomp does, so apologies if this is ignorant. Trent -- Trent Mick trentm at activestate.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
It seems the best thing might be to modify the PCBuild8 build process so the output binaries are in the ../PCBuild' directory - this way distutils and others continue to work fine. Does that sound reasonable? I think Kristjan will have to say a word here: I think he just likes it the way it is right now. That would rather suggest that build_ext needs to be changed. I use this patch in ActivePython to get distutils to find the correct PCbuild dir (see attached). Trent -- Trent Mick trentm at activestate.com --- python/Lib/distutils/command/build_ext.py Tue Mar 13 03:19:35 2007 +++ python/Lib/distutils/command/build_ext.py Tue Apr 17 12:51:26 2007 @@ -176,7 +176,16 @@ # Append the source distribution include and library directories, # this allows distutils on windows to work in the source tree self.include_dirs.append(os.path.join(sys.exec_prefix, 'PC')) -self.library_dirs.append(os.path.join(sys.exec_prefix, 'PCBuild')) +from distutils.msvccompiler import get_build_version +msvc_version = get_build_version() +if msvc_version == 6: +self.library_dirs.append(os.path.join(sys.exec_prefix, 'PC', 'VC6')) +elif 6 msvc_version 8: +self.library_dirs.append(os.path.join(sys.exec_prefix, 'PCbuild')) +elif msvc_version = 8: +self.library_dirs.append(os.path.join(sys.exec_prefix, 'PCbuild8')) +else: +log.warn(unexpected MSVC version: %r, msvc_version) # OS/2 (EMX) doesn't support Debug vs Release builds, but has the # import libraries in its Config subdirectory ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
On 5/23/07, Kristján Valur Jónsson [EMAIL PROTECTED] wrote: Install in the ProgramFiles folder. Only over my dead body. *This* is silly. Bill doesn't think so. And he gets to decide. I mean we do want to play nice, don't we? Nothing installs itself in the root anymore, not since windows 3.1 Maybe installing in the root is not good, but installing to Program Files is just asking for trouble. All sorts of development tools might suddenly break because of that space in the middle of the path and requirement to use quotes around it. I thus usually install things to drive:\Programs. I'm not sure if any packages/programs will break because of that space, but what if some will? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Very well, leaving linux aside, I don't see why this:
/win32mount/trunk/PCbuild/
/x64mount/trunk/PCbuild/
Is any different from
/winmount/trunk/PCBuild/win32
/winmount/trunk/PCBuild/x64
I don't understand this extraordinary reluctance to add a single extra
directory.
The windows build process is different from any other build process, so even
If all the other platforms live one directory higher, why must windows?
It doesn't need to, and reluctance is not wrt. to the proposed new
layout, but wrt. changing the current one. Tons of infrastructure
depends on the files having exactly the names that they have now,
and being located in exactly the locations where they are currently
located. Any change to that, whatever minor, will cause problems
to some people.
So for the change to be made, the advantages must clearly outweigh
the incompatibility of the change in the first place.
I say let's just admit that tools can compile for
more than one target. Let's adapt to it and we will all be happier.
You might be; I will be sad. It comes for a price,
I well understand the benefits, I use it all the time, but the price
still eludes me. how can a different name for the output folder
for a different platform be such a big problem?
When running Python now, I type (after having changed to the source
directory in the cmd.exe window) PCbtabpyttabtab or some
such. To navigate to a subdirectory, I need many more keystrokes.
I find it very painful to invoke python.exe from the
PCbuild8 directory. I don't want that pain to be the default.
Yes, that is correct. Well, there is a CRT .exe redist if you want to deploy
this into the SxS cache, it just has to be run as part of the install process.
But that may or may not be problematic, I don't know.
Microsoft recommends to use the merge module (.msm), and I think this
is what should be done (if feasible).
Why should there be versioning problems with python25.dll? Are there
any past issues with incompatibilities with any python2x.dll release?
Someone could replace the python25.dll that we ship with their own patched
version, thereby gaining backdoor access to the software. The way
windows searches for old style dlls makes this easy. Using the SxS
signed loading scheme, you can protect yourself up to a point from such
attacks. Of course, this doesn't have to be a problem for vanilla
python, we can do this on our own for the patched python25 that we
employ, but it still might be something others could find useful.
I personally think that if hostile users can replace DLLs on your
system, you have bigger problems than SxS installation of pythhonxy.dll,
but perhaps that's just me.
Install in the ProgramFiles folder.
Only over my dead body. *This* is silly.
Bill doesn't think so. And he gets to decide.
He can decide not to give Python the Vista ready logo. If so,
I don't want it. He cannot decide to make the Python installer
to install into a directory with a space in its name by default.
I mean we do want to play nice, don't we?
Nice to users of Python, sure. I would not have said a word if
the standard directory would have been, say \usr\bin. However,
they happened to chose Program Files, making it language dependent,
and putting a space in the name. That space has caused numerous
problems to Python scripts, and I expect changing the default
would cause a lot of problems to end users.
What about the registry?
I don't know about the registry, what is it used for?
For two things, with different importance to different users:
1. File extensions are registered there, e.g. .py and .pyc.
With two binaries installed, the will stomp over each other's
file associations; only one of them can win.
2. Python installs keys under
HL{LM|CU}\Software\Python\PythonCore\version, namely
InstallPath
InstallGroup
PythonPath
Documentation
Modules
For some of these, add-on libraries and applications may modify
these keys, and the interpreter will pick up the changes.
Again, there can be only one installation on the system
owning these settings; two simultaneous installations will
stomp on each other's settings.
64 bit windows already ships with dual versions of some apps, notably
explorer.exe so that shouldn't be a big problem.
The two versions of MSIE actually *are* a big problem, that's
why MS only runs the 32-bit IE, even on Win64 (otherwise, ActiveX
controls downloaded from the net wouldn't work).
Also, while they are both shipped, you can't the two versions of
explorer.exe simultaneously (without trickery), so its far from
simple.
But that's not a problem, is it? Writing silently fails, i.e. it just
won't save the pyc files. Happens all the time on Unix.
It may not silently fail, depending on your user status. An admin might
get a confirmation window, for example.
Can you describe the precise scenario which makes that happen? To my
knowledge, Vista will *not* open a confirmation window when Python
attempts
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
I use this patch in ActivePython to get distutils to find the correct PCbuild dir (see attached). Would you like to commit this to 2.6? (or perhaps 2.5 even?) Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
- Run the appropriate environment setup for the correct compiler. E.g., for the Platform SDK AMD64 compiler and with the current Platform SDK this is: C:\Program Files\Microsoft Platform SDK\SetEnv.Cmd /X64 /RETAIL - Run the solution file with devenv.com (IIRC, devenv.exe doesn't take command-line args) and be sure to pass ing /useenv to pick up the environment changes. (*) set DEVENV_COM=path/to/devenv.com %DEVENV_COM% PCbuild\pcbuild.sln /useenv /build ReleaseAMD64 Yes, that should work equally fine. I haven't looked into what vsextcomp does, so apologies if this is ignorant. It spares you having to setup the environment; it provides a cl.exe wrapper that locates the SDK from the registry, and then invokes the cl.exe in the SDK if necessary. As a consequence, you can still just double-click the solution file, without having to run devenv.exe/com manually. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Very well, leaving linux aside, I don't see why this: /win32mount/trunk/PCbuild/ /x64mount/trunk/PCbuild/ Is any different from /winmount/trunk/PCBuild/win32 /winmount/trunk/PCBuild/x64 In the former case, assuming python is running from the 'trunk' directory, all architectures know how to locate the binary directory - it is always named 'PCBuild'. In the latter case, the directory name depends on the architecture. A number of existing tools already know about the 'PCBuild' directory, so these tools will not need to be taught anything new for x64. I don't understand this extraordinary reluctance to add a single extra directory. I think that this thread has enumerated the concerns fairly well. You may not agree with them, but if you don't understand them it might be worth re-reading Martin's responses. Note that I also understand your concerns and goals - I certainly see where you are coming from - but we have 2 competing goals - work with as many existing, external build tools as possible versus take this opportunity to create a new cross-compile-capable x64 build environment for Windows and let those external tools deal with the breakage. As I mentioned in a previous email, my personal opinion would be swayed by looking externally. Specifically, if we could determine the likelihood of external build processes (eg, mozilla) working unchanged if we stick with 'PCBuild', and if we could determine the cross-compilation strategy being adopted by the external libs we use (zlib, bsddb, etc), I think we could make an informed decision. You might be; I will be sad. It comes for a price, I well understand the benefits, I use it all the time, but the price still eludes me. how can a different name for the output folder for a different platform be such a big problem? Please see above - its not a problem if you think of the PCBuild8 process as the last step in a build process - but often it is not. External tools that use Python (ie, things you try and build after the Python build has completed) are impacted. I understand that you might not use such tools, but they do exist. Why should there be versioning problems with python25.dll? Are there any past issues with incompatibilities with any python2x.dll release? Someone could replace the python25.dll that we ship with their own patched version, thereby gaining backdoor access to the software. The way windows searches for old style dlls makes this easy. Using the SxS signed loading scheme, you can protect yourself up to a point from such attacks. I'm not sure I buy this. If someone has enough access to your machine to change pythonxx.dll, you are pretty screwed already. What about the registry? I don't know about the registry, what is it used for? Please see PC/getpathp.c in Python source tree. However, I agree that there are a number of things we could do to help Python play nicely on Vista. It might help if we can enumerate the specific problems and potential solutions in a more formal way (eg, a Python bug) Cheers, Mark attachment: winmail.dat___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Martin v. Löwis wrote: I use this patch in ActivePython to get distutils to find the correct PCbuild dir (see attached). Would you like to commit this to 2.6? (or perhaps 2.5 even?) Sure, if others think it is a good thing. Will do tomorrow unless I hear a -1 before then. Trent -- Trent Mick trentm at activestate.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Martin v. Löwis wrote: I use this patch in ActivePython to get distutils to find the correct PCbuild dir (see attached). Would you like to commit this to 2.6? (or perhaps 2.5 even?) Sure, if others think it is a good thing. Will do tomorrow unless I hear a -1 before then. I'm not quite a '-1', but am a little confused about where this would leave us. To some extent, this would formalize PCBuild8 and VC6 directories. External tools would then slowly start growing support for these additional directories and the previous benefits of PCBuild is the canonical location appear to vanish. Further, I expect that such a patch would confuse any attempts to manually copy from PCBuild8 into PCBuild, for example (ie, some tools knowing about PCBuild8 while others assume PCBuild is likely to get confusing.) Trent: I assume you use the same source tree for multiple platforms and compilers, meaning that changing these optional build processes to copy from PCBuild8/VC6 into PCBuild would cause pain? If not, do you think that would be a reasonable solution? Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
I'm not quite a '-1', but am a little confused about where this would leave us. To some extent, this would formalize PCBuild8 and VC6 directories. External tools would then slowly start growing support for these additional directories and the previous benefits of PCBuild is the canonical location appear to vanish. Further, I expect that such a patch would confuse any attempts to manually copy from PCBuild8 into PCBuild, for example (ie, some tools knowing about PCBuild8 while others assume PCBuild is likely to get confusing.) Trent: I assume you use the same source tree for multiple platforms and compilers, meaning that changing these optional build processes to copy from PCBuild8/VC6 into PCBuild would cause pain? If not, do you think that would be a reasonable solution? Changing to have bits always in PCbuild would work for me -- i.e. I *don't* build for multiple compilers/platforms in the same tree. Perhaps that is a better solution -- in the long run, anyway. Having the bits always in one dir for whatever the configuration is more akin to the Unix-y configure/make system. Is this something that could work for Python 2.5? Or just 2.6? Long term/aside: Moving to a configure/make build system on Windows, as you proposed in your first email, would be interesting. With MSYS though, not cygwin (a la bsmedberg's new MozillaBuild stuff). I just wish there were an autoconf alternative that wasn't as painful as autoconf. I have a few attempts for my purposes that are written in Python (an obvious bootstrapping problem for building Python itself :). Trent -- Trent Mick trentm at activestate.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
However, I think people ask much too often for a debugging build; in many cases, they could work happily with a release build that supports debugging. Depending on the problem you try to solve, you may or may not need debug information for pythonxy.dll as well, or just for your extension module. While that is true in theory, I often find it is not the case in practice, generally due to the optimizer. Depending on what magic the compiler has done, it can be very difficult to set breakpoints (conditional or otherwise), inspect variable values, etc. It is useful in some cases, but very often I find myself falling back to a debug build after attempting to debug a local release build. I think there are several other cases in distutils to support special cases (e.g. the DISTUTILS_USE_SDK environment variable); if people want to see their setup supported, AND ARE WILLING TO CONTRIBUTE: more power to them. Yes, but I believe its also important to solicit feedback on the best way to achieve their aims. In this particular case, I believe it would have been misguided for me to simply check in whatever was necessary to have distutils work from the PCBuild8 directory. I hope it is clear that I am willing to contribute the outcome of these discussions... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Hi Mark, +1 from me. I think this is simply a bug introduced with the UCS4 patches in Python 2.2. unicodeobject.h already has this code: #ifndef PY_UNICODE_TYPE /* Windows has a usable wchar_t type (unless we're using UCS-4) */ # if defined(MS_WIN32) Py_UNICODE_SIZE == 2 # define HAVE_USABLE_WCHAR_T # define PY_UNICODE_TYPE wchar_t # endif # if defined(Py_UNICODE_WIDE) # define PY_UNICODE_TYPE Py_UCS4 # endif #endif But for some reason, pyconfig.h defines: /* Define as the integral type used for Unicode representation. */ #define PY_UNICODE_TYPE unsigned short /* Define as the size of the unicode type. */ #define Py_UNICODE_SIZE SIZEOF_SHORT /* Define if you have a useable wchar_t type defined in wchar.h; useable means wchar_t must be 16-bit unsigned type. (see Include/unicodeobject.h). */ #if Py_UNICODE_SIZE == 2 #define HAVE_USABLE_WCHAR_T #endif disabling the default settings in the unicodeobject.h. Yes, that does appear strange. The following patch works for me, keeps Python building and appears to solve my problem. Any objections? Looks fine to me. Mark Index: pyconfig.h === --- pyconfig.h (revision 55487) +++ pyconfig.h (working copy) @@ -491,22 +491,13 @@ /* Define if you want to have a Unicode type. */ #define Py_USING_UNICODE -/* Define as the integral type used for Unicode representation. */ -#define PY_UNICODE_TYPE unsigned short - /* Define as the size of the unicode type. */ -#define Py_UNICODE_SIZE SIZEOF_SHORT +/* This is enough for unicodeobject.h to do the right thing on Windows. */ +#define Py_UNICODE_SIZE 2 -/* Define if you have a useable wchar_t type defined in wchar.h; useable - means wchar_t must be 16-bit unsigned type. (see - Include/unicodeobject.h). */ -#if Py_UNICODE_SIZE == 2 -#define HAVE_USABLE_WCHAR_T - /* Define to indicate that the Python Unicode representation can be passed as-is to Win32 Wide API. */ #define Py_WIN_WIDE_FILENAMES -#endif /* Use Python's own small-block memory-allocator. */ #define WITH_PYMALLOC 1 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/mal%40egenix.com -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 22 2007) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
-Original Message- From: Martin v. Löwis [mailto:[EMAIL PROTECTED] Sent: Tuesday, May 22, 2007 05:33 To: Steve Holden Cc: Kristján Valur Jónsson; Mark Hammond; [EMAIL PROTECTED]; python-dev@python.org Subject: Re: Adventures with x64, VS7 and VS8 on Windows Addressing only the issues of PCBuild8 and 64-bit architectures, I have tried to establish free buildbot support for the 64-bit architectures without any real success. Should the PSF be considering paying for infrastructure that will support 64-bit build reporting? You can bring it up to the board, of course. However, given that all other buildbot machines were contributed by volunteers, the fact that nobody volunteers to contribute buildbot machines for PCbuild8 indicates that there is not a lot of interest in that build infrastructure. Most people just install whatever it is that they are offered to download. For me, the most compelling reason to provide the new builds is the 15% performance benefit. If there are no technical and corporate difficulties, such as firewalls and security, I am sure that CCP can provide a VisualStudio2005 buildbot for our needs. Wasn't there some issue that each buildbot can only provide a single build? Here is a place where cross-compilation comes into its own again. Kristjan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
-Original Message- Nobody proposed to ditch cross-compilation. I very much like cross-compilation, I do all Itanium and AMD64 in cross-compilation. I just want the *file structure* of the output to be the very same for all architectures, meaning that they can't coexist in a single source directory. Surely there are differences between architectures? PC uses MSI after all. Why can't linux be under trunk/linux and pc 86 under trunk/pcbuild8/win32PGO and 64 under trunk/pcbuild8/x64pgo? We shouldn't let bad tools keep us from new ways of doing things, rather we should fix and update the tools. No, you use two checkouts, of course. That´s just silly. And two visual studios open, and edit the file in two places too? I say let's just admit that tools can compile for more than one target. Let's adapt to it and we will all be happier. No matter what the next compiler is (VS 2005 or VS 2007/2008), it's still *a lot* of work until the VS 2005 build can be used for releasing Python. For example, there is no support for the SxS installation of msvcr8.dll, using the MSI merge module. Well, there is some work, which is why I am proposing the Europython sprint to do it. But we are almost there. And that work will be useful for Orcas, when that comes along. And btw, there is no need to install the msvcr8.dll. We can distribute them as a private assembly. then they (and the manifest) exist in the same directory as python2x.dll. This is a supported distribution mode and how we distribute the runtime with EVE. Not sure whether anything really is needed. Python works fine on Vista. If you are an administrator. A limited user will have problems installing it and then running it. 1) supplying python.dll as a Side By Side assembly What would that improve? Well, it should reduce dll-hell problems of applications that ship with python2x.dll. You ship with and link to your own and tested dll. We have some concerns here, for example, now that we are moving away from embedding python in our blue.dll and using python25.dll directly, that this exposes a vulnerability to the integrity of the software. 2) Changing python install locations To conform with Windows rules and get a Vista approved logo. Install in the ProgramFiles folder. Just as C does. Ah, and this also means that we could install both 32 bit and 64 bit versions, another plus. 3) Supporting shadow libraries, where .pyc files end up in a different hierarchy from the .py files. (useful for many things beside VISTA) Yes, and people had written PEPs for it which failed due to lack of follow up. Interesting. We are definitely interested in that. You see, Someone installs a game or accounting software using vista. He then runs as a limited user. Python insists on saving its .pyc files in the installation folder, but this is not something that is permitted on Vista. 4) Signing the python dlls and executables Hmm. Again, easing the installation experience for vista users. So that they don't get a red box about an unknown application requiring administrator privileges. Sure, and have they reported problems with Python on Vista (problems specific to Vista?) Certainly. We are working on them, of course. Chiefly, we have had to change where we save all kinds of temporary files. And we have to be careful that all our .py files ship as .pyc files. And we are versioning, and signing all executables, providing user level manifests and fixing up the install processes to be more compliant. I am just saying that this is something that a standard python distro might want to do too. Kristjan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
-Original Message- It seems the best thing might be to modify the PCBuild8 build process so the output binaries are in the ../PCBuild' directory - this way distutils and others continue to work fine. Does that sound reasonable? I think Kristjan will have to say a word here: I think he just likes it the way it is right now. That would rather suggest that build_ext needs to be changed. Someone mentioned the idea to have a bat file do it. I like that idea. There is already a build.bat file which will build whatever configuration you choose (platform and configuration). Extending it to subsequently copy the resulting binaries up one level is easy. Perhaps this is an acceptable compromise? Kristjan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Someone mentioned the idea to have a bat file do it. I like that idea. There is already a build.bat file which will build whatever configuration you choose (platform and configuration). Extending it to subsequently copy the resulting binaries up one level is easy. Perhaps this is an acceptable compromise? Sure - that will work for me. I'll check this out and contact you by private mail for further guidance. Cheers, Mark attachment: winmail.dat___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
I'd be happy to install Windows and the latest VisualStudio on a 64-bit VMware image. I just can't be responsible for day-to-day administration of the buildslave; Windows requires too much attention for me to do that. Thanks for the offer. Perhaps Kristjan is interested in setting up a buildslave on such an installation. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
While that is true in theory, I often find it is not the case in practice, generally due to the optimizer. Depending on what magic the compiler has done, it can be very difficult to set breakpoints (conditional or otherwise), inspect variable values, etc. It is useful in some cases, but very often I find myself falling back to a debug build after attempting to debug a local release build. What I typically do is to disable optimization in a release build. It then essentially becomes a debug build, except that _DEBUG is not defined, so that it doesn't incorporate the debug CRT. This, in turn, means that you can mix such a debug build readily with non-debug binaries. I think there are several other cases in distutils to support special cases (e.g. the DISTUTILS_USE_SDK environment variable); if people want to see their setup supported, AND ARE WILLING TO CONTRIBUTE: more power to them. Yes, but I believe its also important to solicit feedback on the best way to achieve their aims. In this particular case, I believe it would have been misguided for me to simply check in whatever was necessary to have distutils work from the PCBuild8 directory. I hope it is clear that I am willing to contribute the outcome of these discussions... Sure - all understood, and all fine. It's certainly good to build consensus first, but in cases of should we support that borderline case of build setup as long as I'm willing to maintain it, the answer should always be yes (*), irrespective of the merits of the specific proposal. Heck, we would even accept patches to support HP-UX better :-) Regards, Martin (*) assuming it doesn't break anything P.S. I think there have been cases where I haven't been that supportive wrt. build issues. However, in this case I really feel go ahead. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Surely there are differences between architectures? PC uses MSI after all. Why can't linux be under trunk/linux and pc 86 under trunk/pcbuild8/win32PGO and 64 under trunk/pcbuild8/x64pgo? That couldn't work for me. I try avoid building on a network drive, and with local drives, I just can't have a Windows build and a Linux build on the same checkout - they live on separate file systems, after all (Linux on ext3, Windows on NTFS, with multi-boot switching between them). That´s just silly. And two visual studios open, and edit the file in two places too? I have about 10 checkouts of Python, on different machines, with no problems. I don't feel silly doing so. I don't *use* them simultaneously, of course - I cannot work on two architectures simultaneously, anyway. I say let's just admit that tools can compile for more than one target. Let's adapt to it and we will all be happier. You might be; I will be sad. It comes for a price, and with little benefit. Disk space is cheaper than my time to fight build processes. And btw, there is no need to install the msvcr8.dll. We can distribute them as a private assembly. then they (and the manifest) exist in the same directory as python2x.dll. Yes, but then python2x.dll goes into system32, and so will msvcr8.dll, no? Not sure whether anything really is needed. Python works fine on Vista. If you are an administrator. A limited user will have problems installing it and then running it. Is there a bug report for that? 1) supplying python.dll as a Side By Side assembly What would that improve? Well, it should reduce dll-hell problems of applications that ship with python2x.dll. You ship with and link to your own and tested dll. We have some concerns here, for example, now that we are moving away from embedding python in our blue.dll and using python25.dll directly, that this exposes a vulnerability to the integrity of the software. Why should there be versioning problems with python25.dll? Are there any past issues with incompatibilities with any python2x.dll release? 2) Changing python install locations To conform with Windows rules and get a Vista approved logo. Install in the ProgramFiles folder. Only over my dead body. *This* is silly. Just as C does. Ah, and this also means that we could install both 32 bit and 64 bit versions, another plus. What about the registry? Interesting. We are definitely interested in that. You see, Someone installs a game or accounting software using vista. He then runs as a limited user. Python insists on saving its .pyc files in the installation folder, but this is not something that is permitted on Vista. But that's not a problem, is it? Writing silently fails, i.e. it just won't save the pyc files. Happens all the time on Unix. Sure, and have they reported problems with Python on Vista (problems specific to Vista?) Certainly. We are working on them, of course. But, of course, they have not been reported. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
If there are no technical and corporate difficulties, such as firewalls and security, I am sure that CCP can provide a VisualStudio2005 buildbot for our needs. Wasn't there some issue that each buildbot can only provide a single build? Yes, but you can have multiple buildbot slaves on a single machine. They all create separate checkouts. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Hi Martin, You are likely doing something wrong: a) I assume it's VS 7.1 (i.e. VS.NET 2003); VS 2002 is not supported at all b) you probably didn't install vsextcomp, but you should. In fact, you don't need all of it, but you do need the cl.exe and link.exe wrappers it comes with - they dispatch to the proper tools from the SDK c) in case it isn't clear: you also need an AMD64 compiler, e.g. from the platform SDK. Unfortunately, Microsoft keeps changing the registry settings for the SDK, so vsextcomp only knows about some selected SDKs. If that causes a problem, please let me know. I'm using the full-blown VS.NET 2003, as given to a number of python-dev people by Microsoft a number of years ago. This appears to come with the SDK and a 64bit compiler. I'm guessing vsextcomp doesn't use the Visual Studio 'ReleaseAMD64' configuration - would it be OK for me to check in changes to the PCBuild projects for this configuration? This is an issue to be discussed for Python 2.6. I'm personally hesitant to have the official build infrastructure deviate from the layout that has been in-use for so many years, as a lot of things depend on it. Yes, I agree - although I consider x64 new enough that an opportunity exists to set a new 'standard'. However, if most 'external' build processes will not otherwise need to change for a 64bit environment, then I agree that nothing should change in Python's layout. Right - PCbuild8 should not get formalized. It probably should continue to be maintained. So is there something we can do to make distutils play better with binaries built from PCBuild8, even though it is considered temporary? It seems the best thing might be to modify the PCBuild8 build process so the output binaries are in the ../PCBuild' directory - this way distutils and others continue to work fine. Does that sound reasonable? For 2.6, the first question to answer is: what compiler should it use? I would personally like to see Python skip VS 2005 altogether, as it will be soon superceded by Orcas. Unfortunately, it's unclear how long Microsoft will need to release Orcas (and also, when Python 2.6 will be released), so I would like to defer that question by a few months. I've no objection to that - but I'd like to help keep the pain to a minimum for people who find themselves trying to build 64bit extensions in the meantime. Anecdotally, VS8 is the compiler most people start trying to use for this (quite possibly because that is what they already have handy). See above; I disagree. First, multiple architectures only means x86, AMD64, and Itanium, and I would like to drop official Itanium binaries from 2.6 (even though they could continue to be supported in the build process). Then, even if the Python build itself support multiple simultaneous architectures, the extension modules don't all (correct me if I'm wrong). Yes, I agree that it is unlikely to work in practice - at least for a number of years as the external libs and extensions catch up. * Wide characters in VS8: PC/pyconfig.h defines PY_UNICODE_TYPE as 'unsigned short', which corresponds with both 'WCHAR' and 'wchar' in previous compiler versions. VS8 defines this as wchar_t, which I'm struggling to find a formal definition for beyond being 2 bytes. In C or in C++? In C++, wchar_t is a builtin type, just like short, int, long. So there is no further formal definition. This was in C++, but the problem was really WCHAR, as used by much of the win32 API. I'd rather make it a platform-specific definition (for platform=Windows API). Correct me if I'm wrong, but isn't wchar_t also available in VS 2003 (and even in VC6?). And doesn't it have the right definition in all these compilers? hrm - as above, I'm more concerned with the definition of WCHAR - which means my problem is related more to the Platform SDK version rather than the compiler. This is unfortunate - on one hand we do consider 'platform=Windows API', and WCHAR is very much an API concept. I'll need to dig some more into this, but at least I know I'm not wasting my time :) * Finally, as something from left-field which may well take 12 months or more to pull off - but would there be any interest to moving the Windows build process to a cygwin environment based on the existing autoconf scripts? What compiler would you use there? I very much like using the VS debugger when developing on Windows, so that capability should not go away. You would use whatever compiler the autoconf toolset found. Recent versions know enough about MSVC for simple projects. Many people would need to take care that their environment pointed at the correct compiler - especially the person building releases. But assuming MSVC was found and had the appropriate switches passed, there would be no impact on the ability to use Visual Studio as a debugging environment. Thanks, Mark
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
On 2007-05-21 12:30, Kristján Valur Jónsson wrote: [Py_UNICODE being #defined as unsigned short on Windows] I'd rather make it a platform-specific definition (for platform=Windows API). Correct me if I'm wrong, but isn't wchar_t also available in VS 2003 (and even in VC6?). And doesn't it have the right definition in all these compilers? So +1 for setting Py_UNICODE to wchar_t on Windows. Yes. Btw, in previous visual studio versions, wchar_t was not treated as a builtin type by default, but rather as synonymous with unsighed short. Now the default is that it is, and this causes some semantic differences and incompatibilities of the type seen. +1 from me. If think this is simply a bug introduced with the UCS4 patches in Python 2.2. unicodeobject.h already has this code: #ifndef PY_UNICODE_TYPE /* Windows has a usable wchar_t type (unless we're using UCS-4) */ # if defined(MS_WIN32) Py_UNICODE_SIZE == 2 # define HAVE_USABLE_WCHAR_T # define PY_UNICODE_TYPE wchar_t # endif # if defined(Py_UNICODE_WIDE) # define PY_UNICODE_TYPE Py_UCS4 # endif #endif But for some reason, pyconfig.h defines: /* Define as the integral type used for Unicode representation. */ #define PY_UNICODE_TYPE unsigned short /* Define as the size of the unicode type. */ #define Py_UNICODE_SIZE SIZEOF_SHORT /* Define if you have a useable wchar_t type defined in wchar.h; useable means wchar_t must be 16-bit unsigned type. (see Include/unicodeobject.h). */ #if Py_UNICODE_SIZE == 2 #define HAVE_USABLE_WCHAR_T #endif disabling the default settings in the unicodeobject.h. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 21 2007) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
-Original Message- This was in C++, but the problem was really WCHAR, as used by much of the win32 API. I'd rather make it a platform-specific definition (for platform=Windows API). Correct me if I'm wrong, but isn't wchar_t also available in VS 2003 (and even in VC6?). And doesn't it have the right definition in all these compilers? hrm - as above, I'm more concerned with the definition of WCHAR - which means my problem is related more to the Platform SDK version rather than the compiler. This is unfortunate - on one hand we do consider 'platform=Windows API', and WCHAR is very much an API concept. I'll need to dig some more into this, but at least I know I'm not wasting my time :) Mark, your problem may be related to a setting in the c/c++ - language tab in the settings, where treat wchar_t as a builtin type default has changed. I recommend that we do treat it as a builtin, but the VS2003 default was no and the 2005 is yes. Could this be contributing to your problem? Kristjan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Hi Kristján, First of all, I have put some work into pcbuild8 recently and it works well. It does! There are a few issues though, notably with distutils (and as mentioned before, any other tools what may assume PCBuild - see below) You quoting Martin: I don't find the need to have separate object directories convincing: For building the Win32/Win64 binaries, I have separate checkouts *anyway*, since all the add-on libraries would have to support multi-arch builds, but I think they don't. No they don't, but that doesn't mean that you need different checkouts for python, only the others. Anyway, this is indeed something I'd like to see addressed. I don't think we should ditch cross-compilation. While I agree with you, Martin's point about the dependant libraries and tools is valid, and may defeat this goal. In the short term, we should research how some of these other projects are approaching x64 - we may also find that this helps with any autoconf work we choose to do. Ultimately, the person releasing the official binaries gets to say how this works (based on how they actually get it to work) I would personally like to see Python skip VS 2005 altogether, as it will be soon superceded by Orcas. Unfortunately, it's unclear how long Microsoft will need to release Orcas (and also, when Python 2.6 will be released), so I would like to defer that question by a few months. I think this is a bit unrealistic. Here we are in the middle of 2007, VS2005 has just got SP1, and VS2003 is still the official compiler. PCBuild8 is ready, it just needs a little bit of extra love and buildbots to make us able to release PGO versions of x86 and x64. Given the delay for getting even this far, waiting for Orcas and then someone to create PCBuild9, and then getting it up and running and so on will mean waiting another two years. I don't believe there was any suggestion that Python 2.6 would wait for a compiler release from Microsoft. Before we talk about Vista and while I have your attention wink, some final questions relating to PCBuild8. Regardless of the ultimate layout for x64, what do you think about having PCBuid8 put the binaries into the PCBuild directory, and thus (theoretically) letting such a directory work with distutils and otherwise as a fully functional Python installation? I am not familiar with the msi packaging process at all. But here is something we should start to consider: VISTA support. This could mean some of: 1) supplying python.dll as a Side By Side assembly Yes, this is something pywin32 is going to face. The hack of copying python*.dll into the 'system' directory - necessary for COM - is (sensibly!) no longer working. One thing at a time though... 2) Changing python install locations 3) Supporting shadow libraries, where .pyc files end up in a different hierarchy from the .py files. (useful for many things beside VISTA) 4) Signing the python dlls and executables 5) Providing user level manifests. And dragging distutils back op topic, having bdist_wininst supply a manifest that indicates escalation is required appears necessary. Vista adoption is going very fast. We see 10% of our users have moved to vista and rising. ack - I'm yet to try a 32 bit version, but my Vista-x64 box isn't proving very reliable yet. It *is* very pretty and cute though :) I'm surprised to see 10% unless your users are skewed towards early-adopters though, but I'm in no position to refute it! Cheers, Mark attachment: winmail.dat___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
I'm using the full-blown VS.NET 2003, as given to a number of python-dev people by Microsoft a number of years ago. This appears to come with the SDK and a 64bit compiler. Not sure what it makes it appear to you that way - it doesn't. VS.NET 2003 is x86 only I'm guessing vsextcomp doesn't use the Visual Studio 'ReleaseAMD64' configuration - would it be OK for me to check in changes to the PCBuild projects for this configuration? Please don't. It exclusively relies on vsextcomp, and is only useful if you have that infrastructure installed. See for yourself: it uses the /USE_CL:MS_OPTERON command line switch, which isn't a Microsoft invention (but instead invented by Peter Tröger). So is there something we can do to make distutils play better with binaries built from PCBuild8, even though it is considered temporary? In what way better? It already supports them just fine, AFAICT. I guess you are referring to the support for building extensions on top of a source tree installation. I doubt that is used that often (but I understand you are using it). It seems the best thing might be to modify the PCBuild8 build process so the output binaries are in the ../PCBuild' directory - this way distutils and others continue to work fine. Does that sound reasonable? I think Kristjan will have to say a word here: I think he just likes it the way it is right now. That would rather suggest that build_ext needs to be changed. I've no objection to that - but I'd like to help keep the pain to a minimum for people who find themselves trying to build 64bit extensions in the meantime. I recommend that those people install the official binaries. Why do you need to build the binaries from source, if all you want is to build extensions? In C or in C++? In C++, wchar_t is a builtin type, just like short, int, long. So there is no further formal definition. This was in C++, but the problem was really WCHAR, as used by much of the win32 API. But in C, WCHAR should not be a problem (and I would like to see explicit source code and explicit compiler error message to be proven wrong). * Finally, as something from left-field which may well take 12 months or more to pull off - but would there be any interest to moving the Windows build process to a cygwin environment based on the existing autoconf scripts? What compiler would you use there? I very much like using the VS debugger when developing on Windows, so that capability should not go away. You would use whatever compiler the autoconf toolset found. Recent versions know enough about MSVC for simple projects. Many people would need to take care that their environment pointed at the correct compiler - especially the person building releases. But assuming MSVC was found and had the appropriate switches passed, there would be no impact on the ability to use Visual Studio as a debugging environment. I doubt that could work in practice. You will have to rewrite everything to make it pass the correct command line switches. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
I don't find the need to have separate object directories convincing: For building the Win32/Win64 binaries, I have separate checkouts *anyway*, since all the add-on libraries would have to support multi-arch builds, but I think they don't. No they don't, but that doesn't mean that you need different checkouts for python, only the others. Anyway, this is indeed something I'd like to see addressed. I don't think we should ditch cross-compilation. Nobody proposed to ditch cross-compilation. I very much like cross-compilation, I do all Itanium and AMD64 in cross-compilation. I just want the *file structure* of the output to be the very same for all architectures, meaning that they can't coexist in a single source directory. It should simplify a lot of stuff, including buildbot setup and so on (if we get the buildbot infrastructure to support it). It is also very cumbersome, if you are working on a project, to have the binaries all end up in the same place. Doing interactive work on python, I frequently compile both the 32 bit and 64 bit versions for testing and it would be downright silly to have to rebuild everything every time. No, you use two checkouts, of course. I think this is a bit unrealistic. Here we are in the middle of 2007, VS2005 has just got SP1, and VS2003 is still the official compiler. PCBuild8 is ready, it just needs a little bit of extra love and buildbots to make us able to release PGO versions of x86 and x64. No matter what the next compiler is (VS 2005 or VS 2007/2008), it's still *a lot* of work until the VS 2005 build can be used for releasing Python. For example, there is no support for the SxS installation of msvcr8.dll, using the MSI merge module. Given the delay for getting even this far, waiting for Orcas and then someone to create PCBuild9, and then getting it up and running and so on will mean waiting another two years. No. I would expect that either the PCbuild or PCbuild8 project files can be used with little changes to build using VS9. I just tried, and it works reasonably well. I am not familiar with the msi packaging process at all. But here is something we should start to consider: VISTA support. This could mean some of: Not sure whether anything really is needed. Python works fine on Vista. 1) supplying python.dll as a Side By Side assembly What would that improve? 2) Changing python install locations Why? 3) Supporting shadow libraries, where .pyc files end up in a different hierarchy from the .py files. (useful for many things beside VISTA) Yes, and people had written PEPs for it which failed due to lack of follow up. 4) Signing the python dlls and executables Hmm. 5) Providing user level manifests. Vista adoption is going very fast. We see 10% of our users have moved to vista and rising. Sure, and have they reported problems with Python on Vista (problems specific to Vista?) Yes. Btw, in previous visual studio versions, wchar_t was not treated as a builtin type by default, but rather as synonymous with unsighed short. Now the default is that it is, and this causes some semantic differences and incompatibilities of the type seen. C or C++? According to the standards, it ought to be a builtin, primitive type in C++, and a typedef in C. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Hi Marc-Andre, +1 from me. If think this is simply a bug introduced with the UCS4 patches in Python 2.2. unicodeobject.h already has this code: #ifndef PY_UNICODE_TYPE /* Windows has a usable wchar_t type (unless we're using UCS-4) */ # if defined(MS_WIN32) Py_UNICODE_SIZE == 2 # define HAVE_USABLE_WCHAR_T # define PY_UNICODE_TYPE wchar_t # endif # if defined(Py_UNICODE_WIDE) # define PY_UNICODE_TYPE Py_UCS4 # endif #endif But for some reason, pyconfig.h defines: /* Define as the integral type used for Unicode representation. */ #define PY_UNICODE_TYPE unsigned short /* Define as the size of the unicode type. */ #define Py_UNICODE_SIZE SIZEOF_SHORT /* Define if you have a useable wchar_t type defined in wchar.h; useable means wchar_t must be 16-bit unsigned type. (see Include/unicodeobject.h). */ #if Py_UNICODE_SIZE == 2 #define HAVE_USABLE_WCHAR_T #endif disabling the default settings in the unicodeobject.h. Yes, that does appear strange. The following patch works for me, keeps Python building and appears to solve my problem. Any objections? Mark Index: pyconfig.h === --- pyconfig.h (revision 55487) +++ pyconfig.h (working copy) @@ -491,22 +491,13 @@ /* Define if you want to have a Unicode type. */ #define Py_USING_UNICODE -/* Define as the integral type used for Unicode representation. */ -#define PY_UNICODE_TYPE unsigned short - /* Define as the size of the unicode type. */ -#define Py_UNICODE_SIZE SIZEOF_SHORT +/* This is enough for unicodeobject.h to do the right thing on Windows. */ +#define Py_UNICODE_SIZE 2 -/* Define if you have a useable wchar_t type defined in wchar.h; useable - means wchar_t must be 16-bit unsigned type. (see - Include/unicodeobject.h). */ -#if Py_UNICODE_SIZE == 2 -#define HAVE_USABLE_WCHAR_T - /* Define to indicate that the Python Unicode representation can be passed as-is to Win32 Wide API. */ #define Py_WIN_WIDE_FILENAMES -#endif /* Use Python's own small-block memory-allocator. */ #define WITH_PYMALLOC 1 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Hi Martin, I'm using the full-blown VS.NET 2003, as given to a number of python-dev people by Microsoft a number of years ago. This appears to come with the SDK and a 64bit compiler. Not sure what it makes it appear to you that way - it doesn't. VS.NET 2003 is x86 only Yes - I was confused by finding an x64 configuration option, and this looking very similar to VC8. My apologies for the confusion. So is there something we can do to make distutils play better with binaries built from PCBuild8, even though it is considered temporary? In what way better? It already supports them just fine, AFAICT. I guess you are referring to the support for building extensions on top of a source tree installation. I doubt that is used that often (but I understand you are using it). Yes, that is correct. I agree it will be rarely used by Python user's, but believe it is a common scenario for people who maintain extensions or libraries, particularly those who want debugging builds. It seems the best thing might be to modify the PCBuild8 build process so the output binaries are in the ../PCBuild' directory - this way distutils and others continue to work fine. Does that sound reasonable? I think Kristjan will have to say a word here: I think he just likes it the way it is right now. That would rather suggest that build_ext needs to be changed. So this means PCBuild8 does indeed get formalized into distutils? I'm happy to live with it if it lets me work, but it seems contrary to our emails yesterday. It would also mean the PCBuild8 environment will not work with external build processes that assume the standard layout, but that really isn't something I'm willing to run the pydev gauntlet for at the current time. I've no objection to that - but I'd like to help keep the pain to a minimum for people who find themselves trying to build 64bit extensions in the meantime. I recommend that those people install the official binaries. Why do you need to build the binaries from source, if all you want is to build extensions? Let's assume that people have a valid reason to build from source - it really is quite common in the open source world. The meta-question then becomes is it reasonable to expect people be able to build extensions from a tree built with VC8, in the same way they can with VS7?. I think you are suggesting we do want to support this, but I want to be clear. In C or in C++? In C++, wchar_t is a builtin type, just like short, int, long. So there is no further formal definition. This was in C++, but the problem was really WCHAR, as used by much of the win32 API. But in C, WCHAR should not be a problem (and I would like to see explicit source code and explicit compiler error message to be proven wrong). Please see my other mail to Kristjan - at this stage I can only reproduce it in C++ on VC8. C++ on VC7 and C on VC8 both work fine. Please let me know if thde code snippet I pasted or the compiler error aren't suitable. I've stopped further investigation since there seems support for a change that makes the problem go away. * Finally, as something from left-field which may well take 12 months or more to pull off - but would there be any interest to moving the Windows build process to a cygwin environment based on the existing autoconf scripts? What compiler would you use there? I very much like using the VS debugger when developing on Windows, so that capability should not go away. You would use whatever compiler the autoconf toolset found. Recent versions know enough about MSVC for simple projects. Many people would need to take care that their environment pointed at the correct compiler - especially the person building releases. But assuming MSVC was found and had the appropriate switches passed, there would be no impact on the ability to use Visual Studio as a debugging environment. I doubt that could work in practice. You will have to rewrite everything to make it pass the correct command line switches. 'have to rewrite everything' sounds a little pessimistic, but that's fine with me - consider this idea dropped. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Kristján Valur Jónsson wrote: First of all, I have put some work into pcbuild8 recently and it works well. I am trying to drum up momentum for work on PCBuild8 next europython. See http://wiki.python.org/moin/EuroPython2007Sprints -Original Message- From: [EMAIL PROTECTED] I don't find the need to have separate object directories convincing: For building the Win32/Win64 binaries, I have separate checkouts *anyway*, since all the add-on libraries would have to support multi-arch builds, but I think they don't. No they don't, but that doesn't mean that you need different checkouts for python, only the others. Anyway, this is indeed something I'd like to see addressed. I don't think we should ditch cross-compilation. It should simplify a lot of stuff, including buildbot setup and so on (if we get the buildbot infrastructure to support it). It is also very cumbersome, if you are working on a project, to have the binaries all end up in the same place. Doing interactive work on python, I frequently compile both the 32 bit and 64 bit versions for testing and it would be downright silly to have to rebuild everything every time. I would personally like to see Python skip VS 2005 altogether, as it will be soon superceded by Orcas. Unfortunately, it's unclear how long Microsoft will need to release Orcas (and also, when Python 2.6 will be released), so I would like to defer that question by a few months. I think this is a bit unrealistic. Here we are in the middle of 2007, VS2005 has just got SP1, and VS2003 is still the official compiler. PCBuild8 is ready, it just needs a little bit of extra love and buildbots to make us able to release PGO versions of x86 and x64. Given the delay for getting even this far, waiting for Orcas and then someone to create PCBuild9, and then getting it up and running and so on will mean waiting another two years. Addressing only the issues of PCBuild8 and 64-bit architectures, I have tried to establish free buildbot support for the 64-bit architectures without any real success. Should the PSF be considering paying for infrastructure that will support 64-bit build reporting? [...] regards Steve -- Steve Holden+1 571 484 6266 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden -- Asciimercial - Get on the web: Blog, lens and tag your way to fame!! holdenweb.blogspot.comsquidoo.com/pythonology tagged items: del.icio.us/steve.holden/python All these services currently offer free registration! -- Thank You for Reading ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Yes, that is correct. I agree it will be rarely used by Python user's, but believe it is a common scenario for people who maintain extensions or libraries, particularly those who want debugging builds. Ah, debugging builds. It's true that PCbuild does not support them for AMD64, and it's also true that you need such a build if you want the debug CRT. However, I think people ask much too often for a debugging build; in many cases, they could work happily with a release build that supports debugging. Depending on the problem you try to solve, you may or may not need debug information for pythonxy.dll as well, or just for your extension module. I'd like to repeat an offer that I have made several times in the past: if somebody contributes a patch to msi.py which packs all PDB files (and whatever you else you need) into a ZIP file (or whatever else works) from the release build, then I could happily release PDB files along with the actual installer files. (I would not like to release the PDB files *in* the installer files, as I expect they would blow up the msi size significantly). I think Kristjan will have to say a word here: I think he just likes it the way it is right now. That would rather suggest that build_ext needs to be changed. So this means PCBuild8 does indeed get formalized into distutils? I'm happy to live with it if it lets me work, but it seems contrary to our emails yesterday. It would mean that - I'm willing to compromise to make Kristjan happy (he contributed PCbuild8, so he has to decide what changes to it are acceptable and which aren't). A middle ground might be an addition build step (e.g. as a .bat file) which copies all result files also into PCbuild. Let's assume that people have a valid reason to build from source - it really is quite common in the open source world. The meta-question then becomes is it reasonable to expect people be able to build extensions from a tree built with VC8, in the same way they can with VS7?. I think you are suggesting we do want to support this, but I want to be clear. As long as people are willing to maintain it - why not? It's not official in the sense that if it breaks, nobody might fix it, but it doesn't hurt to have it (AFAICT). Also, it not being official might mean that we are not obliged to provide backwards compatibility for it, so it can go away without notice (e.g. when/if PCbuild8 is dropped). The same could be done for PC/VC6, if anybody cared to contribute and maintain it. I think there are several other cases in distutils to support special cases (e.g. the DISTUTILS_USE_SDK environment variable); if people want to see their setup supported, AND ARE WILLING TO CONTRIBUTE: more power to them. Please see my other mail to Kristjan - at this stage I can only reproduce it in C++ on VC8. C++ on VC7 and C on VC8 both work fine. Please let me know if thde code snippet I pasted or the compiler error aren't suitable. It's fine. I readily believe that it causes problems in C++. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Addressing only the issues of PCBuild8 and 64-bit architectures, I have tried to establish free buildbot support for the 64-bit architectures without any real success. Should the PSF be considering paying for infrastructure that will support 64-bit build reporting? You can bring it up to the board, of course. However, given that all other buildbot machines were contributed by volunteers, the fact that nobody volunteers to contribute buildbot machines for PCbuild8 indicates that there is not a lot of interest in that build infrastructure. Note that there *are* 64-bit buildbot slaves, e.g. for AMD64 (contributed by Neal Norwitz), Alpha (contributed by Neal Norwitz), Itanium (contributed by Matthias Klose, offering a machine from the Debian project), and PPC64 (likewise). These machines all run Linux or Tru64, and (to my understanding) serve other purposes as well, making the buildbot slave just a minor detail. It's an unfortunate fact of life that Microsoft Windows does not support multi-tasking multi-user workloads that well, so the Windows buildbot slave are (to my knowledge) typically dedicated machines (except for Tim's machine, which is just disconnected from the master when he doesn't feel like running buildbot jobs). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
On Tue, May 22, 2007 at 07:32:42AM +0200, Martin v. L?wis wrote: - Addressing only the issues of PCBuild8 and 64-bit architectures, I - have tried to establish free buildbot support for the 64-bit - architectures without any real success. - - Should the PSF be considering paying for infrastructure that will - support 64-bit build reporting? - - You can bring it up to the board, of course. However, given that - all other buildbot machines were contributed by volunteers, the - fact that nobody volunteers to contribute buildbot machines for - PCbuild8 indicates that there is not a lot of interest in that - build infrastructure. - - Note that there *are* 64-bit buildbot slaves, e.g. for AMD64 - (contributed by Neal Norwitz), Alpha (contributed by Neal - Norwitz), Itanium (contributed by Matthias Klose, offering - a machine from the Debian project), and PPC64 (likewise). - - These machines all run Linux or Tru64, and (to my understanding) - serve other purposes as well, making the buildbot slave - just a minor detail. - - It's an unfortunate fact of life that Microsoft Windows does not - support multi-tasking multi-user workloads that well, so the - Windows buildbot slave are (to my knowledge) typically dedicated - machines (except for Tim's machine, which is just disconnected - from the master when he doesn't feel like running buildbot - jobs). I haven't really been listening to this conversation, so forgive me if this isn't relevant, but: I'd be happy to install Windows and the latest VisualStudio on a 64-bit VMware image. I just can't be responsible for day-to-day administration of the buildslave; Windows requires too much attention for me to do that. cheers, --titus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Adventures with x64, VS7 and VS8 on Windows
Hi all, I hope the cross-post is appropriate. I've started playing with getting the pywin32 extensions building under the AMD64 architecture. I started building with Visual Studio 8 (it was what I had handy) and I struck a few issues relating to the compiler version that I thought worth sharing. * In trying to build x64 from a 32bit VS7 (ie, cross-compiling via the PCBuild directory), the python.exe project fails with: pythoncore fatal error LNK1112: module machine type 'X86' conflicts with target machine type 'AMD64' is this a known issue, or am I doing something wrong? * The PCBuild8 project files appear to work without modification (I only tried native compilation here though, not a cross-compile) - however, unlike the PCBuild directory, they place all binaries in a 'PCBuild8/x64' directory. While this means that its possible to build for multiple architectures from the same source tree, it makes life harder for tools like 'distutils' - eg, distutils already knows about the 'PCBuild' directory, but it knows nothing about either PCBuild8 or PCBuild8/x64. A number of other build processes also know to look inside a PCBuild directory (eg, Mozilla), so instead of formalizing PCBuild8, I think we should merge PCBuild8 into PCBuild. This could mean PCBuild/vs7 and PCBuild/vs8 directories with the project files, but binaries would still be generated in the 'PCBuild' (or PCBuild/x64) directory. This would mean the same tree isn't capable of hosting 2 builds from different VS compilers, but I think that is reasonable (if it's a problem, just have multiple source directories). I understand that PCBuild8 is not official, but in the assumption that future versions of Python will use a compiler later than VS7, it makes sense to me to clean this up now - what are others opinions on this? * Re the x64 directory used by the PCBuild8 process. IMO, it makes sense to generate x64 binaries to their own directory - my expectation is that cross-compiling between platforms is a reasonable use-case, and we should support multiple achitectures for the same compiler version. This would mean formalizing the x64 directory in both 'PCBuild' and distutils, and leaving other external build processes to update as they support x64 builds. Does this make sense? Would this fatally break other scripts used for packaging (eg, the MSI framework)? * Wide characters in VS8: PC/pyconfig.h defines PY_UNICODE_TYPE as 'unsigned short', which corresponds with both 'WCHAR' and 'wchar' in previous compiler versions. VS8 defines this as wchar_t, which I'm struggling to find a formal definition for beyond being 2 bytes. My problem is that code which assumes a 'Py_UNICODE *' could be used in place of a 'WCHAR *' now fails. I believe the intent on Windows has always been PyUNICODE == 'native unicode' - should PC/pyconfig.h reflect this (ie, should pyconfig.h grow a version specific definition of PyUNICODE as wchar_t)? * Finally, as something from left-field which may well take 12 months or more to pull off - but would there be any interest to moving the Windows build process to a cygwin environment based on the existing autoconf scripts? I know a couple of projects are doing this successfully, including Mozilla, so it has precendent. It does impose a greater burden on people trying to build on Windows, but I'd suggest that in recent times, many people who are likely to want to build Python on Windows are already likely to have a cygwin environment. Simpler mingw builds and nuking MSVC specific build stuff are among the advantages this would bring. It is not worth adding this as yet another windows build option - so IMO it is only worth progressing with if it became the blessed build process for windows - if there is support for this, I'll work on it as the opportunity presents itself... I'm (obviously) only suggesting we do this on the trunk and am happy to make all agreed changes - but I welcome all suggestions or critisisms of this endeavour... Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
* In trying to build x64 from a 32bit VS7 (ie, cross-compiling via the PCBuild directory), the python.exe project fails with: pythoncore fatal error LNK1112: module machine type 'X86' conflicts with target machine type 'AMD64' is this a known issue, or am I doing something wrong? You are likely doing something wrong: a) I assume it's VS 7.1 (i.e. VS.NET 2003); VS 2002 is not supported at all b) you probably didn't install vsextcomp, but you should. In fact, you don't need all of it, but you do need the cl.exe and link.exe wrappers it comes with - they dispatch to the proper tools from the SDK c) in case it isn't clear: you also need an AMD64 compiler, e.g. from the platform SDK. Unfortunately, Microsoft keeps changing the registry settings for the SDK, so vsextcomp only knows about some selected SDKs. If that causes a problem, please let me know. * The PCBuild8 project files appear to work without modification (I only tried native compilation here though, not a cross-compile) - however, unlike the PCBuild directory, they place all binaries in a 'PCBuild8/x64' directory. While this means that its possible to build for multiple architectures from the same source tree, it makes life harder for tools like 'distutils' - eg, distutils already knows about the 'PCBuild' directory, but it knows nothing about either PCBuild8 or PCBuild8/x64. This is an issue to be discussed for Python 2.6. I'm personally hesitant to have the official build infrastructure deviate from the layout that has been in-use for so many years, as a lot of things depend on it. I don't find the need to have separate object directories convincing: For building the Win32/Win64 binaries, I have separate checkouts *anyway*, since all the add-on libraries would have to support multi-arch builds, but I think they don't. A number of other build processes also know to look inside a PCBuild directory (eg, Mozilla), so instead of formalizing PCBuild8, I think we should merge PCBuild8 into PCBuild. Right - PCbuild8 should not get formalized. It probably should continue to be maintained. For 2.6, the first question to answer is: what compiler should it use? I would personally like to see Python skip VS 2005 altogether, as it will be soon superceded by Orcas. Unfortunately, it's unclear how long Microsoft will need to release Orcas (and also, when Python 2.6 will be released), so I would like to defer that question by a few months. I understand that PCBuild8 is not official, but in the assumption that future versions of Python will use a compiler later than VS7, it makes sense to me to clean this up now - what are others opinions on this? Not official really only means not used to build the official binaries - just like PC/VC6. It's still (somewhat) maintained. As for cleaning it up - see above. I would *really* like to skip VS 2005 altogether, as I expect that soon after we decide to use VS 2005, Microsoft will replace it with the next release, stop supporting VS 2005, take the free compiler off the next, and so on (just like they did for VS 2003, soon after we decided to use it for 2.5). * Re the x64 directory used by the PCBuild8 process. IMO, it makes sense to generate x64 binaries to their own directory - my expectation is that cross-compiling between platforms is a reasonable use-case, and we should support multiple achitectures for the same compiler version. See above; I disagree. First, multiple architectures only means x86, AMD64, and Itanium, and I would like to drop official Itanium binaries from 2.6 (even though they could continue to be supported in the build process). Then, even if the Python build itself support multiple simultaneous architectures, the extension modules don't all (correct me if I'm wrong). This would mean formalizing the x64 directory in both 'PCBuild' and distutils, and leaving other external build processes to update as they support x64 builds. Does this make sense? Would this fatally break other scripts used for packaging (eg, the MSI framework)? The MSI packaging would need to be changed, certainly. It currently detects the architecture it needs to package by looking at the file type of python.exe; that would have to be changed to give it an explicit parameter what architecture to package, or have it package all architectures it can find. * Wide characters in VS8: PC/pyconfig.h defines PY_UNICODE_TYPE as 'unsigned short', which corresponds with both 'WCHAR' and 'wchar' in previous compiler versions. VS8 defines this as wchar_t, which I'm struggling to find a formal definition for beyond being 2 bytes. In C or in C++? In C++, wchar_t is a builtin type, just like short, int, long. So there is no further formal definition. In C (including C99), wchar_t ought to be defined in stddef.h. My problem is that code which assumes a 'Py_UNICODE *' could be used in place of a 'WCHAR *' now fails. I believe the intent on Windows has always
