[issue16895] Batch file to mimic 'make' on Windows
Zachary Ware added the comment: Still relevant, but the current patch is overkill (Brian was right, it's just way more batch than is healthy to check in). Things are changing rapidly, though; I'll close it as postponed for now, with the expectation of reopening someday. -- assignee: - zach.ware components: +Build resolution: - postponed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Mark Lawrence added the comment: Is this still relevant or has it been overtaken by other work from Zach or Steve? -- nosy: +BreamoreBoy, steve.dower versions: +Python 3.5 -Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Changes by Brian Curtin br...@python.org: -- nosy: -brian.curtin ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Changes by Mark Lawrence breamore...@yahoo.co.uk: -- nosy: -BreamoreBoy ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Changes by Mark Lawrence breamore...@yahoo.co.uk: -- nosy: +BreamoreBoy ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Richard Oudkerk added the comment: I can't say I know enough about batch files to understand much of the code, but a few notes: Windows XP does not have the command where which you use -- Python 3.4 will still support XP. Except perhaps for looping I would prefer to get rid of the use of goto. The fact that some goto targets end in exit /b ... make it very confusing as to where exit /b will return control. The initial pushd is matched by various popd's which are scattered over hundreds of lines (including one in :usage). I think it would be better to keep matching pushd/popd reasonably close together. For instance, I think you could do something like ... pushd %~dp0 call :main ... popd exit /b :main ... exit /b It would also be helpful if the end of the subroutines were marked with a comment like rem end :foo -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Brian Curtin added the comment: Can't this just be a Python script? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Richard Oudkerk added the comment: Can't this just be a Python script? That would cause bootstrap issues for people who do not already have python installed. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Zachary Ware added the comment: Richard Oudkerk added the comment: I can't say I know enough about batch files to understand much of the code, but a few notes: Windows XP does not have the command where which you use -- Python 3.4 will still support XP. Oh, that is an issue. I don't have an XP machine to test on anymore, thank you for that catch. I found a workaround on StackOverflow that looks short enough to be usable instead. Except perhaps for looping I would prefer to get rid of the use of goto. The fact that some goto targets end in exit /b ... make it very confusing as to where exit /b will return control. The only goto's that are not part of loops are now one near the beginning for the -C option, in the target validation routine to show the usage message and die, and in a couple of routines which use goto no-configure to show a common message and die. I'd rather not have to copy that message every place it is used, but that is an option. Would just adding comments explaining where execution is going and whether it is coming back be sufficient? The initial pushd is matched by various popd's which are scattered over hundreds of lines (including one in :usage). I think it would be better to keep matching pushd/popd reasonably close together. For instance, I think you could do something like ... pushd %~dp0 call :main ... popd exit /b :main ... exit /b Fair enough, I can change that. I tried to keep the matches to the initial pushd to a minimum, but perhaps there are a couple more I can eliminate. It would also be helpful if the end of the subroutines were marked with a comment like rem end :foo Easy enough, consider it done :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Brian Curtin added the comment: Don't we already require an existing Python to build some of the third-party stuff, e.g., OpenSSL? I don't think the bootstrapping issue holds that much weight. Adding some huge batch script that maybe one or two people even know how to modify is a much higher cost than just having someone install Python. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Zachary Ware added the comment: Brian Curtin added the comment: Don't we already require an existing Python to build some of the third-party stuff, e.g., OpenSSL? Only for building a 64-bit Python on 32-bit Windows. Otherwise, OpenSSL is built using the just-built interpreter. I don't think the bootstrapping issue holds that much weight. Adding some huge batch script that maybe one or two people even know how to modify is a much higher cost than just having someone install Python. Fair enough, but even when Python is installed, there's still the issue of whether it will be findable on PATH, whether .py is in PATHEXT, what version of Python is installed (and which one is on PATH), etc. etc... However, you've made me think; perhaps configure.bat could build a minimal Python that can then be used for a make.py. I'll do some experimenting and see what I can come up with. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Zachary Ware added the comment: Here's a new version of the patch, which is a major rewrite. Among the changes: - Switch from using 'goto' to execute the right subroutine to 'call' and end every subroutine with 'exit /B' (thank you Richard for making me aware of that possibility). This makes the :end label useless, so it's removed. Every subroutine now expects to start in the root of the source tree, and is expected to return there before exiting. - Changed the argument parsing magic to a different flavor of magic. Setting variables on the command line is now much more like it is using Unix make, no more required double quotes around the whole assignment, and you can now specify multiple targets in a single command. This method can be quite a bit slower if you give a huge command, as it steps through the command line a character at a time (on the order of several seconds if you're appending something to PATH, for instance. The delay is not noticeable if you're just giving targets), but at least all the magic is self-contained. - Add '--no-externals' and '--interactive' options to configure.bat to ignore the presence or absence of the external libs and to allow questions to be asked of the user, respectively. I've waffled back and forth on whether it is better for the '--interactive' option to be on or off by default, currently it is off. - On 64 bit machines, default to Win32 platform if vcvarsx86_amd64.bat can't be found. - Give some nicer output just about everywhere. This includes making configure.bat die early in case %VS100COMNTOOLS% is not defined with a message to install MSVC++ 2010 Express, a warning if NASM is not available, and tagging all output with the name of the script. Also, add a message at the end of building Python giving the best estimate of the outcome. - Another part of giving a nice message at the end of building Python: if the expected executable file exists at the end, exit code 0 is returned. This is for the buildbots, so that even if the build had errors, the tests are still run so we can see what is actually broken. This ties in with the new '--no-externals' option on configure.bat, which would allow a buildbot to build with '--no-externals' and still be able to test everything else. - List and use specific exit codes. If anyone has advice on better numbers to use (more standard numbers, perhaps), please let me know :). - Use absolute relative paths (paths that always start with %~dp0, but may include '..') to avoid some directory changing magic just to make a relative path point to the right place. - 'clobber' target builds the clean target of Win32 configuration, then x64 config if the environment can be set up to do so. - 'importlib' will clean up after itself if there wasn't a build already done. - Some refactoring and cleanup to put things in places that make more sense and look better. This includes major changes to :external and :tcltk to use a couple of loops to be more DRY. - Change Tools/scripts/run_tests.py (used by the test targets in make.bat and Makefile) to use subprocess.call instead of os.execv on win32. Since execv creates a new process on Windows, control was handed back to the console as soon as the execv call was made, but the test output still came to the terminal so the prompt was lost in the test output and did not automatically come up at the end of the test. Using subprocess.call keeps control in the parent python(_d) process and doesn't release back to the console until the test is done. - Added better documentation of make.bat usage to PCbuild\readme.txt, and took out unrelated changes to that file (which will be posted to another issue). Everything works as expected for me, but I do only have a 32bit machine available to me currently. I hope to be able to test on a 64bit machine soon, but have no guarantees. For anyone testing this, I would suggest to run the command prompt $+%PROMPT% before testing; this will add a + to the beginning of your prompt for every dir on the pushd/popd stack. Neither configure.bat nor make.bat should ever remove dirs from that stack, and should not leave any extras when they're done (unless forcibly killed in the middle of running). -- Added file: http://bugs.python.org/file30351/win_configure-make.v2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Zachary Ware added the comment: You seem to end your subroutines (or whatever they are called) using goto end rather than exit /b. Since popd follows the end label, does this mean that you get a popd after calling each subroutine? Yes. Is this intended and can it cause unmatched pushd/popd-s? It is intended and it can, but so far I haven't run into any trouble with it. However, I have realized that I haven't done any testing with some pushd's already on the stack... I'll do some more looking into that, and may have to implement either some form of callback labels or counting pushd's to properly popd. (I am not familiar with writing batch files.) Also, I think 32 bit builds should be the default. Many people with 64 bit Windows are using Visual Studio Express which only has 32 bit support. Fair point. I suppose I had been assuming that the error message from not finding the proper vcvars*.bat file would be an indication to pass '--without-64-bit' to configure.bat, but it is much nicer to just fall back to 32 bit. Also, it would be nice to just implode from the start if there is no compiler available. Next version of the patch will do both of these things in configure.bat; quit with an error message if %VS100COMNTOOLS% is not defined, and check for the existence of %VS100COMNTOOLS%\..\..\VC\bin\x86_amd64\vcvarsx86_amd64.bat before setting %PY_PLATFORM% to x64. Thanks for the comments. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Richard Oudkerk added the comment: You seem to end your subroutines (or whatever they are called) using goto end rather than exit /b. Since popd follows the end label, does this mean that you get a popd after calling each subroutine? Is this intended and can it cause unmatched pushd/popd-s? (I am not familiar with writing batch files.) Also, I think 32 bit builds should be the default. Many people with 64 bit Windows are using Visual Studio Express which only has 32 bit support. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Zachary Ware added the comment: I was rather off about kill_python, wasn't I? Anyway, I've finally gotten another version put together that I'd like to get comments on. Major differences between this patch and the previous one, in no particular order: - Add a 'configure.bat' script to increase the parallel with Unix building. Currently only has 2 options, --with-pydebug and --with(out)-64-bit, which affect how PY_PLATFORM and PY_CONFIGURATION are set in make.bat, which allows for building any combination of Debug, Release, 32 bit, and 64 bit with make.bat. - Add 'win-config.dat' to .hgignore; this file stores configure.bat's options for make.bat's use. - Update documentation: PCbuild/readme.txt gets significant updates, including some that apply without make.bat and which I can split out into a separate patch if someone wants to commit those separately. Doc/using/windows.rst also gets a small update which actually just goes with one of the PCbuild/readme.txt updates and isn't affected by make.bat. - make.bat is now stand-alone; it does not rely on any other batch scripts except configure.bat (namely Tools/buildbot/*). - Renamed ready target to external, as promised. - Reimplemented test target to match the Unix implementation. Also added buildbottest target, which matches Unix implementation. - Added separate tcltk target to build tcltk and copy the built .dlls to the appropriate place for the built Python to find them. - Added all alias to build target (to match Unix). build calls :external on its own, and calls :tcltk if it can't find the appropriate tcl/tk .dlls. - Added clobber target, which does its best to obliterate everything make.bat can create. The only way I've thought of to be more thorough is to actually delete the entire contents of the PCbuild directory and do an ``hg revert -C PCbuild`` to get it back, but that does have the possibility to be more destructive than desired. - Added variables to the top of the file to set the versions of the external projects. This means there's only one place to update when an external is updated, and for easy testing one could do something akin to ``make OPENSSLVERSION=openssl-1.0.1e`` - Added msi target to perform the task of Tools/buildbot/buildmsi.bat. Note that this one is hard for me to test, as buildmsi.bat does not work properly for me. However, the msi target works as well or better than buildmsi.bat for me, and should work assuming msi.py can do what it is meant to. I wonder, though; does anybody use buildmsi.bat? I noticed that it has information in it hard-coded for Python 2.6a3... Several things have gone through an iteration or two since the last patch; I'll try to go through and add some comments on Rietveld myself. Since make.bat is now stand-alone, if it is accepted (and it is deemed desirable to do so), the Windows buildbots could be converted to use make.bat instead of Tools/buildbot/*, and that directory could actually be removed. This would also make it easy to set up a buildbot that tests a Release configuration of Python. All comments welcome :) -- Added file: http://bugs.python.org/file29667/win_configure-make.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Richard Oudkerk added the comment: +1 To use Tools/builbot/*.bat doesn't the current directory have to be the main directory of the repository? Then I see no point in the -C argument: just set the correct directory automatically. I think make.bat should also support creation of non-debug builds. (Maybe have targets release and debug?) Tools/buildbot/build*.bat already calls external.bat and clean.bat. This currently makes the ready target unnecessary. However, I don't think build should be calling clean.bat (or external.bat). Perhaps you should just inline the necessary parts of Tools/buildbot/build*.bat. -- nosy: +sbt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Zachary Ware added the comment: Richard Oudkerk added the comment: +1 Thank you :) To use Tools/builbot/*.bat doesn't the current directory have to be the main directory of the repository? Then I see no point in the -C argument: just set the correct directory automatically. That is true, and it does already automatically set the correct directory. The point of the -C option is for Doc/make.bat--from the root dir, call make -C Doc html, and make cd's into Doc, and there calls make.bat html, and the html docs are made. I didn't just hardwire the option to -C Doc even though that's the only place that can use -C, just in case there is someday another make.bat floating around the repository. The original version I posted just had a doc target that called Doc/make.bat with supplied arguments, but forcing the use of the UNIXy command makes documenting how to do things easier: just one command on all platforms. I think make.bat should also support creation of non-debug builds. (Maybe have targets release and debug?) I agree. I've had a thought for this; what about adding a configure.bat that takes a few of the applicable options from the UNIX configure script, and sets some environment variables (and/or writes out a win-config.dat) for make.bat's use? Tools/buildbot/build*.bat already calls external.bat and clean.bat. This currently makes the ready target unnecessary. I thought the same initially, but I realized that it is useful for building using the VS GUI--clone the repo, call make ready, then open pcbuild.sln in Visual Studio and you're ready to build everything there. However, I don't think build should be calling clean.bat (or external.bat). Perhaps you should just inline the necessary parts of Tools/buildbot/build*.bat. I've thought about this too, and am thinking that it would make a certain amount of sense to inline the entirety of Tools/buildbot. Make everything possible from just make.bat, and eventually migrate the buildbots to use it, and then the Tools/buildbot directory can go away alltogether. This would also make doing release vs debug easier. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Terry J. Reedy added the comment: One must run external.bat if one is to subsequently build from the vs gui with *most* of the external dependencies. I think the command to run it should be 'external', not 'ready'. 'External' means get the external dependencies and anyone who has run external.bat will know what it means. 'Ready' does not mean anything in particular. Just curious: Why do the buildbots run make clean before re-compiling? I seems like lots of extra work to re-compile things that are up to date? What does running 'kill-python before re-building python do? I have not seen it mentioned in the in the devguide or pcbuild/readme. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Zachary Ware added the comment: Terry J. Reedy added the comment: One must run external.bat if one is to subsequently build from the vs gui with *most* of the external dependencies. I think the command to run it should be 'external', not 'ready'. 'External' means get the external dependencies and anyone who has run external.bat will know what it means. 'Ready' does not mean anything in particular. Fair point. Thinking about it again after some sleep, I agree. Next version will revert to 'external' rather than 'ready'. Just curious: Why do the buildbots run make clean before re-compiling? I seems like lots of extra work to re-compile things that are up to date? I think the idea is to make sure everything is fresh, and avoid phantom problems from things that shouldn't cause problems, but do. What does running 'kill-python before re-building python do? I have not seen it mentioned in the in the devguide or pcbuild/readme. I believe it's what actually performs the cleaning process. Tools/buildbot/clean.bat builds kill_python.exe and then runs it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Richard Oudkerk added the comment: What does running 'kill-python before re-building python do? I have not seen it mentioned in the in the devguide or pcbuild/readme. It kills any currently running python(_d).exe processes. This is because Windows does not allow program or library files to be removed or overwritten while they are being used, potentially causing compilation failures. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Terry J. Reedy added the comment: Thanks, clean and kill make sense for unattended build-bots. For interactive use, when 2 minutes rebuilding is a big deal, clean is a last resort. Let's just make sure we do not somehow suggest that it needs to be done routinely. Or to put it another way, do document that 'build' is intended for fail-safer unattended buildbots and not necessarily for interactive human use. And also that kill does what a human would normally do by closing windows on the task bar, or, as a last resort, with task manager. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Terry J. Reedy added the comment: +1 Add make external[-64] to runs Tools/buildbot/external[-amd64].bat and explain this in the devguide (if not already) rather than buried in the opaque PCBuild/readme. Even after reading build.bat, I do not understand what is the different net effect of that versus just double clicking pcbuild.sln and building. So some explanation in the devguide would be needed. -- nosy: +brian.curtin, terry.reedy, tim.golden ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16895] Batch file to mimic 'make' on Windows
Zachary Ware added the comment: Here's an updated version in the form of a patch. Features include: - Ensure things are done in the right directory with pushd/popd - Allow setting environment variables (for the life of the script) with: make test PYTHON=..\PCbuild\python.exe - Emulate UNIX's `make -C dir` (cd dir; call make.bat %*, essentially) - Choose x86 or x64 with -64 command line switch - Intelligently choose an interpreter for `make patchcheck` and `make test` - Available targets are: build, clean, patchcheck, ready, test. ready is implemented by Tools/buildbot/external[-amd64].bat; make ready sounds better and possibly more accurate than make external to me. - Create a convenience python.bat script, which calls the newly built interpreter with all supplied arguments. Created by :build, deleted by :clean (and added to .hgignore) Everything seems to work ok for me, but my testing platforms are limited to a single Win7 laptop, currently. This issue is also somewhat constrained by issue 17202; it is supposed to work without any change to .hgeol, but that can't be guaranteed. -- keywords: +patch Added file: http://bugs.python.org/file29363/win_make.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16895 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
