Re: [PATCH] [cygwin|mingw]: Add cross-compile support to cwrapper (take 6)
Den 2010-08-27 00:27 skrev Roumen Petrov: Ralf Wildenhues wrote: Hi Charles, [SNIP] + func_wine_to_win32_path_result=$1 + if test -n $1; then +# Unfortunately, winepath does not exit with a non-zero +# error code, so we are forced to check the contents of +# stdout. On the other hand, if the command is not +# found, the shell will set an exit code of 127 and print +# *an error message* to stdout. So we must check for both +# error code of zero AND non-empty stdout, which explains +# the odd construction: Starting from wine 1.3.1 wine path always output paths: I have been reading the git source of winepath, and it definitely has code paths which output an empty result when certain things fail (or rather, a single newline for every path/file it fails to convert), so saying that it always output paths is not entirely correct. Lets wine is correctly configured (Z: drive is linked to the file system root): $ cd $WINEPREFIX/dosdevices $ winepath -w `pwd` Z:\%WINEPREFIX_CONVERTED_TO_BACKSLASHES%\dosdevices Now lets remove link: $ rm z: $ winepath -w `pwd` \\?\unix\%WINEPREFIX_CONVERTED_TO_BACKSLASHES%\dosdevices So sed should remove leading //?/unix This failures are probably comping from inside wine (no mention of //?/unix or \\?\unix in the winepath source), so probably won't be seen as failure by winepath. But, if you silently remove \\?\unix, then you'll end up with a path that is not complete with a drive letter. I think any \\?\unix prefix should be filed as a failure... Peter, are you reading this? Looks like a TODO item for automake/lib/compile. ;-) Yes, I'm reading this :-) Patches to compile (and ar-lib) to follow when the dust settles... Cheers, Peter
Re: [PATCH] [cygwin|mingw]: Add cross-compile support to cwrapper (take 6)
Den 2010-08-27 08:25 skrev Peter Rosin: Den 2010-08-27 00:27 skrev Roumen Petrov: Lets wine is correctly configured (Z: drive is linked to the file system root): $ cd $WINEPREFIX/dosdevices $ winepath -w `pwd` Z:\%WINEPREFIX_CONVERTED_TO_BACKSLASHES%\dosdevices Now lets remove link: $ rm z: $ winepath -w `pwd` \\?\unix\%WINEPREFIX_CONVERTED_TO_BACKSLASHES%\dosdevices So sed should remove leading //?/unix This failures are probably comping from inside wine (no mention of This failure is probably coming (fat fingers today...) //?/unix or \\?\unix in the winepath source), so probably won't be seen as failure by winepath. But, if you silently remove \\?\unix, then you'll end up with a path that is not complete with a drive letter. I think any \\?\unix prefix should be filed as a failure... Hmmm, I dug deeper into the wine source and wine_unix_to_nt_file_name in the dlls/ntdll:path.c file seems to sometimes prefix the path with \??\unix (with two question marks), so you seem to have mistyped the above (and not copy-pasted an actual session as I originally thought). It also seems we can forget about the forward slash variant? But I'm not regularly running wine, so someone who does should chime in with verification. Cheers, Peter
Re: [PATCH] [cygwin|mingw]: Add cross-compile support to cwrapper (take 6)
* Charles Wilson wrote on Fri, Aug 27, 2010 at 06:35:31PM CEST:
It appears the problem is that the mdemo{2}_static.exe tests don't work,
when mdemo{2}-conf.test is used. They DO work when mdemo-static.test is
used (there is no mdemo2-static.test).
Now, this is a little disturbing, since my -dlpreopen was supposed to
have fixed this;
Would be good to fix, yes. On *nix, I'd be suggesting git bisect now,
no idea if you can stand the wait on w32 though. At least you should be
able to speed it up significantly with something like
make check TESTSUITEFLAGS=-V TESTS=all the mdemo ones
in the bisection script, and with some luck, such a script can run
unsupervised ...
linux-mingw cross
==
linux-mingw (old tests): 2 of 100 FAIL, 6 SKIP
FAIL: tests/demo-hardcode.test
FAIL: tests/depdemo-relink.test
Don't know if these are regressions or not; will recheck without
this patch.
The real bug here is that the relink tests (both of them, actually)
should be skipped when cross-compiling, because you have no business
trying to execute the $host executable.
I kind of agree. Can we make it so that it is tested, but when the test
programs don't behave as expected, and we cross-compile, we SKIP instead
of FAIL? Or is even trying to execute the broken programs a sin, in
which case we should SKIP outright? Thanks.
Okay, so I rebuilt my linux-cygwin compiler after asking for help on
cyg...@. This one does NOT have sysroot support (one thing at a
time...) but it DOES work. I was able to build not just hello world in
C++, but also to rebuild cygwin itself, and the new DLL worked fine.
So, re-running the linux-cygwin tests, I got:
linux-cygwin (old tests): 2 of 91 FAIL, 33 SKIP
FAIL: tests/demo-hardcode.test
FAIL: tests/demo-relink.test
linux-cygwin (new tests): 59 as expected. 58 skipped.
Three notes:
1) With *this* compiler, there are no regressions with this patch. I
got the same results both with and without this patch.
Cool.
2) The failures on linux-cygwin in the old testsuite are similar to the
failures on linux-mingw in the old testsuite (not the same;
linux-mingw fails depdemo-relink; linux-cygwin fails demo-relink).
3) EVERY new test that FAILED when using my broken compiler was
actually SKIPPED when using the new working one. So, oddly, with the
broken compiler, the testsuite seems to be confused as to whether the
build is a crossbuild or not.
However, I won't be looking in to this further until after I rebuild my
toolchain (again) with sysroot support; probably next week.
OK.
So, to sum up all of these test runs: it does not appear that ANY of the
failures experienced are regressions, or are due to anything associated
with THIS patch, on any of the platforms or configurations I have tested.
Great.
I still need to do the cygwin-mingw (lie) case, but I think I will
save that until after I resolve the code comments.
OK.
Thanks!
Ralf
Re: [PATCH] [cygwin|mingw]: Add cross-compile support to cwrapper (take 6)
Another questions regarding this patch: Do you know whether all of the conversion functions are idempotent (f(f(x)) = f(x))? IOW, when the user passes names already converted, do we do the right things in all cases? Thanks, Ralf
Re: [PATCH] [cygwin|mingw]: Add cross-compile support to cwrapper (take 6)
On 8/26/2010 4:18 PM, Ralf Wildenhues wrote:
--- a/Makefile.am
+++ b/Makefile.am
@@ -528,7 +528,8 @@ TESTS_ENVIRONMENT = MAKE=$(MAKE) CC=$(CC)
CFLAGS=$(CFLAGS) \
CXX=$(CXX) CXXFLAGS=$(CXXFLAGS) CXXCPP=$(CXXCPP) \
F77=$(F77) FFLAGS=$(FFLAGS) \
FC=$(FC) FCFLAGS=$(FCFLAGS) \
-GCJ=$(GCJ) GCJFLAGS=$(GCJFLAGS)
+GCJ=$(GCJ) GCJFLAGS=$(GCJFLAGS) \
+lt_cv_to_host_path_cmd=$(to_host_path_cmd)
I don't understand why this change should be necessary. In your
testing, you describe that most setups set a correct to_host_path_cmd
themselves. For the other(s), you already describe
export lt_cv_to_host_path_cmd=override
./configure ...
make all check
That should be sufficient without the above patch, no?
Well, IF you *export* that value before running configure, AND run the
tests from the same shell in which that value is exported, then sure,
that is all you need. (I think. It's been over two years since I wrote
this patch...)
However, we need to ensure that the tests work even if you run them from
a different shell that the one you configured with. I realize we don't
tend to do this for *all* possible cache values that the end-user might
override, but this one (and the one Peter added) are special, because
the documentation is going to talk about them explicitly. We're
actually going to *recommend* that certain classes of users override
these particular cache values. That makes them different from all the
other lt_cv_* variables someone might override.
It's best to avoid surprises, in that case.
Anyway, I reviewed all of the past discussion, and I found where this
was introduced: it was the addendum to version 3:
http://lists.gnu.org/archive/html/libtool-patches/2009-01/msg00238.html
Now, while I admitted uncertainty back then about the Makefile.am
change, I recorded no other notes about the testsuite.at changes or why
I thought they were necessary. I do remember originally implementing it
without those two changes, and then having to add first the
Makefile.am change, and then the testsuite.at change, before I could get
things to work. But, sadly, I no longer remember exactly WHAT the
failure mode was (maybe it had to do with the tests that re-run other
tests, like max_cmd_length?)
I'll try it without both changes (but with lt_cv_* exported in the
current shell) and see what happens.
***QQQ***
(above discussion)
--- a/libltdl/config/ltmain.m4sh
+++ b/libltdl/config/ltmain.m4sh
@@ -2775,166 +2775,595 @@ fi\
}
+
+# PATH CONVERSION HELPER FUNCTIONS #
+
Believe it or not, the GNU Coding Standards ask us to name things file
name that which are files, and path those which are colon-separated
(or $PATH_SEPARATOR-separated). I'll close my eyes now on this issue,
because I see hundreds of instances of it in this patch, so that'd be a
TODO item.
It's not a difficult thing to do, and would be purely mechanical. I can
put that on the queue for right after (or even before) the promised docu
patch.
It needs to happen before the release, because the docu will recommend
that certain users set lt_cv_to_host_path_cmd (ok,
lt_cv_to_host_file_name_cmd) to the name of one of these functions. If
we're going to change the name of the functions, we better do it before
we document them.
+ lt_sed_naive_backslashify='s|*|\\|g;s|/|\\|g;s|\\||g'
Might want to move this variable outside to just initialize it once?
It's global anyway. Maybe even general.m4sh.
It would make sense in general.m4sh. OK.
+func_wine_to_win32_path_tmp=`winepath -w $1 2/dev/null`
+if test $? -eq 0 test -n ${func_wine_to_win32_path_tmp}; then
I'll just note that some shells ((d?)ash 0.2) fail to propagate the exit
status of an assignment. No need to change the code, but users should
have a decent shell for this.
OK. I remember one of the earlier iterations actually invoked the
winepath command twice...I don't remember why, but maybe it was related.
Most shells -- especially on x86 linux where wine would actually be
available -- are decent, tho.
+ func_wine_to_win32_path_result=`$ECHO $func_wine_to_win32_path_tmp |
+$SED -e $lt_sed_naive_backslashify`
+else
+ func_wine_to_win32_path_result=
The way this is coded, correctness relies on the fact that all code
paths that invoke this function do eventually check for non-emptiness
of the result.
Yes. That is documented:
# ARG is the $build path to be converted to win32 format.
# result is available in $func_wine_to_win32_path_result
# result is empty on error (or when arg is empty)
+func_wine_to_win32_pathlist_oldIFS=$IFS
IFS is saved and restored always to the same value in libtool, AFAIK,
so a short variable name should suffice completely.
Ack.
+IFS=:
+for func_wine_to_win32_pathlist_f in $1; do
+ IFS=$func_wine_to_win32_pathlist_oldIFS
+ if test -n
Re: [PATCH] [cygwin|mingw]: Add cross-compile support to cwrapper (take 6)
On 8/26/2010 5:20 PM, Charles Wilson wrote: On 8/26/2010 4:18 PM, Ralf Wildenhues wrote: Then, please just move the new functions where Peter needs them, if they really need moving, that is. I deliberately placed them after func_compile and before func_link, for speed-of-parsing reasons. Obviously libtool is used to compile object code much more often than it is used to link (since every linked result requires one or more, sometimes many more, objects) -- so moving these functions ahead of func_compile will impact speed. How much? Don't know; I'll try to generate some numbers. OTOH, it is absolutely *required* to move them where Peter wants them, since he /must/ use translate some paths if func_compile is to work, with MSVC. So...we have to pay the price regardless. I tested using 'ncurses' -- which conveniently is built using the system installed libtool script, rather than including ltmain.sh etc in its own configury. The times below are for a complete 'make' (after configure has already run) -- so it includes a lot more than just how fast does libtool --mode=compile go. But, the ONLY difference between the two is the relative position of the conversion functions within the libtool script, so any time differences should be attributable solely to that change. Original: real25m3.886s user6m24.620s sys 11m13.787s With the functions moved ahead of func_mode_compile: real24m34.235s user6m30.590s sys 11m23.878s Statistics: 69 executables linked 6 libraries (dlls) linked 654 objects (325 pic, 329 non-pic) So, when compiling about 325 source code files, the new function order cost 5 seconds of user time and 10 seconds of system time, total. That's 1/20th of a second slower per file, under: cygwin-1.7.6-1 bash-3.2.51-24 On a 1.67GHz Core 2 Duo machine running Vista. -- Chuck
Re: [PATCH] [cygwin|mingw]: Add cross-compile support to cwrapper (take 6)
On 8/27/2010 12:54 AM, Ralf Wildenhues wrote:
* Charles Wilson wrote on Thu, Aug 26, 2010 at 11:20:48PM CEST:
Also: I've said this before, but we can't use the m4
function_replace magic because we need to retain the ability for
users to override the choice of $func_to_host_path_cmd. This is
critical for the 'cygwin-mingw (lie)' use case, which is VERY
common for the gcc developers.
Do you need an override at 'make' time or does one at configure time
suffice?
I'm not sure. I thought we needed the override at make time, but this
facility is here specifically to support a use case that *I don't
use*. I just really really really don't want to break the scheme used
by Danny and others when building gcc (even if Danny has now hung up his
gcc mingw maintainer hat).
It's possible that a configure-time-only override is sufficient.
Because if the latter (which I'm assuming), then from my POV the above
is just a statement that function_replace is not good enough for your
needs, not one it cannot be done.
Well, sure: if I thought, at the time, that configure-time-only was
sufficient, I wouldn't have said we can't. I was assuming that we
needed make-time-override capability; IF that is true, then you really
don't want to get into the situation where, at make-time, the user sets
lt_cv_to_host_path_cmd=a_func_we_did_not_emit.
OTOH, if configure-time-only is sufficient, then you really don't need
lt_cv_to_host_path_cmd at all. You just replace the guts of
func_to_host_path() itself. But that's rather a big change, and I
really don't want to go there right now -- especially as I'm not sure of
the answer to the at-configure-time-only or at-make-time question.
I think that question will have to be deferred until all this stuff is
merged, with the make-time-override capability, and then we let folks
(like Danny) get used to it. THEN we ask hey, would it be ok if you
could only modify the file/path conversion schema at configure time,
rather than at make time?
OTOH, as a follow-on patch after 2.2.next, we could implement an
explicit N+M scheme just so that any future extensions -- which I
can't actually foresee -- could hook in scalably.
Oh no, please not code which already sets out as dead code.
No, it wouldn't be dead code. Basically, you'd have
from_$build (actually, we would NOT really implement
any of these; see below)
to_$host== what they are now
There were be TWO variables lt_cv_from_build_cmd and lt_cv_to_host_cmd,
but only one function would ever eval them:
generic_build_to_host() {
$from_build_cmd $1
$to_host_cmd $from_build_cmd_result
generic_build_to_host_result=$to_host_cmd_result
}
But, at present, $from_build_cmd would *always* be assigned to
func_convert_noop.
Everywhere we currently do $to_host_cmd ..., you do
generic_build_to_host ... instead.
It's not dead code -- but it's *useless* code, right now, since with
the weird w32 platforms, you'd be calling $from_build_cmd
(func_convert_noop) uselessly.
If there is
a need for more translations, then maybe, but then I'd suggest there'd
be an effort to get as many of the w32 translators under the same hood
as possible.
I've reworded this discussion and put it into TODO, but don't plan to
address it any more than that. Status quo for now.
--
Chuck
Re: [PATCH] [cygwin|mingw]: Add cross-compile support to cwrapper (take 6)
On 8/27/2010 2:01 PM, Ralf Wildenhues wrote: * Charles Wilson wrote on Fri, Aug 27, 2010 at 06:35:31PM CEST: Now, this is a little disturbing, since my -dlpreopen was supposed to have fixed this; Would be good to fix, yes. On *nix, I'd be suggesting git bisect now, no idea if you can stand the wait on w32 though. At least you should be able to speed it up significantly with something like make check TESTSUITEFLAGS=-V TESTS=all the mdemo ones in the bisection script, and with some luck, such a script can run unsupervised ... The real bug here is that the relink tests (both of them, actually) should be skipped when cross-compiling, because you have no business trying to execute the $host executable. I kind of agree. Can we make it so that it is tested, but when the test programs don't behave as expected, and we cross-compile, we SKIP instead of FAIL? Or is even trying to execute the broken programs a sin, in which case we should SKIP outright? Thanks. I think they should be SKIPPED always. The test actually tries to arrange things so that the executable fails, by hiding its dependent DLLs. Well, on cross, we fail automatically 'cause we can't run the $host exe, so we PASS that part of the test -- but for the wrong reason. And, because the output from the failure is not what we expect...sometimes our failure results in a SKIP or a FAIL. And sometimes we ARE able to run the $host executable (cygwin-mingw, linux-w32 with binfmt) and things just get really hairy. It's just not reliable. SKIP 'em. -- Chuck
Re: [PATCH] [cygwin|mingw]: Add cross-compile support to cwrapper (take 6)
* Charles Wilson wrote on Fri, Aug 27, 2010 at 10:23:31PM CEST: Original: real25m3.886s user6m24.620s sys 11m13.787s With the functions moved ahead of func_mode_compile: real24m34.235s user6m30.590s sys 11m23.878s So, when compiling about 325 source code files, the new function order cost 5 seconds of user time and 10 seconds of system time, total. That's 1/20th of a second slower per file, under: cygwin-1.7.6-1 bash-3.2.51-24 On a 1.67GHz Core 2 Duo machine running Vista. Yes, but there is a significant speedup in real time!? That makes little sense, unless the system was busy doing other things also, for the first run. 5% sucks a bit, fixing that should be a TODO item. Thanks for doing the measurements, Ralf
Re: [PATCH] [cygwin|mingw]: Add cross-compile support to cwrapper (take 6)
I'm ok with everything in this.
Re: [PATCH] [cygwin|mingw]: Add cross-compile support to cwrapper (take 6)
On 8/27/2010 5:48 PM, Ralf Wildenhues wrote: * Charles Wilson wrote on Fri, Aug 27, 2010 at 10:23:31PM CEST: Original: real25m3.886s user6m24.620s sys 11m13.787s With the functions moved ahead of func_mode_compile: real24m34.235s user6m30.590s sys 11m23.878s Yes, but there is a significant speedup in real time!? That makes little sense, unless the system was busy doing other things also, for the first run. Well, yeah -- it's windows. Who KNOWS what it is doing behind the scenes. Also, I was taking the time to compose all those other emails on the same machine; I've got two cores and was only running make -j1, but I'm not surprised there was some impact on the 'real' numbers. But that should not have affected the user and sys numbers. 5% sucks a bit, fixing that should be a TODO item. It's closer to 3%: user: 6:24 = 384; 6 seconds = 1.5% sys: 11:13 = 673; 10 seconds = 1.5% Not great, and it would be nice to fix -- but not terrible. Also, I expect the impact on REAL operating systems would be less; I seem to recall that along with fork() performance, cygwin is really bad (slow) at parsing shell scripts. If we can do the magic m4 func_to_host guts-replacement, instead of using the indirection variables, that should help. But that is a longer term project. -- Chuck
Re: [PATCH] [cygwin|mingw]: Add cross-compile support to cwrapper (take 6)
Hi Charles, * Charles Wilson wrote on Sat, Aug 28, 2010 at 07:10:25AM CEST: On 8/27/2010 3:47 PM, Ralf Wildenhues wrote: * Charles Wilson wrote on Fri, Aug 27, 2010 at 08:49:24PM CEST: However, once I have finished the requested changes above and the rebasing (plus whatever comes of the four open ***QQQ***uestions), I might ask for a 12 hour halt on updates to master so I can run those tests, if that's ok? Why should you need the halt? If there are new pushs in the meantime, you just merge origin/master into your master, and push. Doesn't that mess up the linear history of commits in the upstream master? Well, yes, if you want to call it messing up, that is. From git's perspective, origin/master and master are not the same branch in your repository. Just that most of the time, you use fast-forward or rebase when getting the changes from the former into the latter. After the push, we want master to look like this, right: + master (head): chuck's lastest commit + the next most recent commit + older commits + ... Several projects seem to require it for some notion of cleanliness, but no, if there is a good reason against that, we don't necessarily want that. Your having tested a particular commit is such a good reason, IMVHO. Not this: + master (head): merge from chuck's local master + more recent master commit + some other master commit +++ (chuck's local master): chuck's latest commit + master commit just before chuck began his final testing + older commits The text in the merge is slightly different: master: merge origin/master recent master commit .. chuck's latest commit master commit just before chuck began his final testing Or am I confused about how git handles a merge on a single (tracking) branch? No, you aren't. Just that I think that's a perfectly acceptable thing in this situation. Cheers, Ralf
