RE: Searching configured and relocated prefix.
-Original Message- From: Ian Lance Taylor [mailto:[EMAIL PROTECTED] Sent: Sunday, July 23, 2006 3:57 PM To: Andrew Pinski Cc: Mark Mitchell; Carlos O'Donell; gcc@gcc.gnu.org Subject: Re: Searching configured and relocated prefix. Andrew Pinski [EMAIL PROTECTED] writes: On Jul 23, 2006, at 10:44 AM, Ian Lance Taylor wrote: Me too. Except now you have suggested that we change the current behavior which you already suggested at the GCC summit we should not do at least not for a couple of release for warning people. You cannot have it both ways, people already depend on this behavior, I am one example. My general point at the gcc summit was not one about changing the compiler, although that was the context in which I raised it. It was that we should have more consideration of the needs of our users, and less consideration of the needs of our developers. As far as I am concerned, the relevant question on this issue is whether anybody other than toolchain developers depends on the current behaviour. I believe the answer is no. Since the current behaviour has various drawbacks which are clearly evident in organizations which use NFS widely, and since moreover I believe that most non-gcc- developers do not exepect the current behaviour, I believe that we should change it. Ian -- All, I'm glad to see this issue finally being addressed. Especially since I've been trying to work around this since Nov. 2002. Especially since I discussed this on this mailing list in the past and Mark agreed that this problem should be fixed, hence the filing of bug report (enhancement) #17621, in Sep. 2004: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=17621 I also strongly agree with Danny Smith's email about this. There are precious few people who build their own compiler. Most people just want to use a compiler, and would rather it be built by other people. Rarely will an installed compiler be in the exact same location as where the compiler was built. And you can just about guarantee that is the case when it comes to Windows because of the compiler having to be built with either Cygwin or MinGW/MSYS. In that scenario, it is absolutely stupid to have the compiler search the built prefix. I'm sorry, but it's stupid for the FSF to assume that the only way a user will use the software is that they build it themselves. There should be an innate ability and a simple and easy method for relocation of the software. Eric Weddington
Re: Searching configured and relocated prefix.
Andrew Pinski wrote: On Jul 14, 2006, at 1:17 PM, Carlos O'Donell wrote: We currently search both the relocated compilers prefix and the originally configured prefix. Should a relocated compiler be searching both directories? Yes because someone might have just relocated the compiler but not the rest of the tools. The main use of this is that untar a common sysroot and use different versions of the compiler which normally would be installed under that common location. There are benefits and costs in either direction. 1. If we search both locations (i.e., for a relocated compiler, search the configured-time prefix and the installation-time prefix), we get the following set of problems: (a) If the configure-time prefix exists (for unrelated reasons) the compiler finds files in the configure-time prefix, even though neither the system administrator or user expects that. (b) If the configure-time prefix does not exist, but is under an NFS mount, the compiler will cause automount traffic, NFS timeouts, etc. (c) In searching for files, the compiler will make a lot of stat calls, measurably slowing down a relocated compiler. 2. If we search only location (i.e., for a relocated compiler, search only the installation-time prefix), we get a different set of issues: (a) As you say, a single sysroot (or other toolchain components) cannot as easily shared across compiler installations. However, I think it's clear that the problems in (1) are more severe than the problems in (2), on several grounds: * The problems in (1) are demonstrably annoying to people; CodeSourcery has several customer complaints about different customers related to this issue. All of (1a), (1b), and (1c) have been reported to us. (1a) is particularly nasty; users got totally incorrect behavior out of the compiler because the compiler was searching a configure-time prefix that happened to contain unrelated files. * The problems in (2) can be worked around by (for example) using a symlink to place the sysroot in both installation prefixes. These are actions that can be taken by system administrators at installation-time; they have no affect on ordinary users. * The problems in (1) are due to an implicit behavior of the compiler that empirically violates the principle of least surprise. If you get a tarball full of binaries, and unpack it in /home/mitchell/install, why would you expect it to search /opt/distributor, /tmp/buildroot, etc.? * No non-GCC compiler searches a configure-time prefix. The only locations relevant are well-known paths (like /usr/include) and the installation-time prefix. So, GCC's model is confusing to users coming from other compilers. This is not a definitive argument, but it should carry weight unless there is some strong argument in favor of GCC's current behavior. * I suspect that the problems in (2) are relative rare while the problems in (1) are relatively common. A lot of users download binary distributions and install them somewhere convenient; relatively few try to do complicated things involving partially shared installations, and those users are probably more expert. -- Mark Mitchell CodeSourcery [EMAIL PROTECTED] (650) 331-3385 x713
Re: Searching configured and relocated prefix.
Mark Mitchell [EMAIL PROTECTED] writes: 1. If we search both locations (i.e., for a relocated compiler, search the configured-time prefix and the installation-time prefix), we get the following set of problems: ... 2. If we search only location (i.e., for a relocated compiler, search only the installation-time prefix), we get a different set of issues: ... However, I think it's clear that the problems in (1) are more severe than the problems in (2), on several grounds: Me too. Ian
Re: Searching configured and relocated prefix.
On Jul 23, 2006, at 10:44 AM, Ian Lance Taylor wrote: However, I think it's clear that the problems in (1) are more severe than the problems in (2), on several grounds: Me too. I actually think the problems with 1 (b) are artificial and should not be taken into account. I actually depend on a common sysroot already and it allows me to do development of a newer compiler much faster and it allows some of our game developers to be able to test a new compiler without having another copy of the SDK installed. 1(a) is not confusing at all, if people read the documentation like they should be doing, it should be clear to them that it looks at both locations. For 1(c), it is not going to slow it down that much, in fact it just one or two extra stat pre original stat which should already be in the stat cache unless 1(b) comes into play but that is a miss-configuration of their systems at that point and not really a GCC issue and should not be treated as such. -- Pinski
Re: Searching configured and relocated prefix.
Andrew Pinski wrote: I actually think the problems with 1 (b) are artificial and should not be taken into account. This is not a hypothetical or artificial issue -- as I said, all three problems I listed have been encountered by real users. I actually depend on a common sysroot already and it allows me to do development of a newer compiler much faster and it allows some of our game developers to be able to test a new compiler without having another copy of the SDK installed. Yes, that's clever. But, you can create a symbolic link to the sysroot from each installation with a single command. And, your installer for third-party developers can do that for you. 1(a) is not confusing at all, if people read the documentation like they should be doing, it should be clear to them that it looks at both locations. Most developers do not start by reading the entire compiler manual -- especially including information about building the compiler itself. The typical software developer probably doesn't have any idea what the configure-time prefix for the compiler might even mean. Their knowledge of compilers is probably that -c, -D, -I, -g, and -O2 are useful command-line options. And, if using an IDE, maybe not even that! And, there's no reason they should need more information than that; there's no reason compiler users should have to have any knowldege about how the compiler is put together. For 1(c), it is not going to slow it down that much, in fact it just one or two extra Sorry, this is just incorrect. It's a significant issued (as much as 25% for some projects) as measured by actual customers in the field. play but that is a miss-configuration of their systems at that point and not really a GCC issue and should not be treated as such. A large class of users (most corporate developers, for example) run on systems they don't administer. We want the compiler to perform well on their systems, if possible. Even developers who do administer their own systems may not be expert administrators; I used to administer my own GNU/Linux box, but I didn't know about most of the options to the kernel or other parts of the system. And, many GCC users are running on Windows, where they have less control. -- Mark Mitchell CodeSourcery [EMAIL PROTECTED] (650) 331-3385 x713
Re: Searching configured and relocated prefix.
Yes, that's clever. But, you can create a symbolic link to the sysroot from each installation with a single command. And, your installer for third-party developers can do that for you. What are the equivalent to symbolic links on Windows and I am not talking about cygwin either? Thanks, Andrew Pinski
Re: Searching configured and relocated prefix.
(1b) If the configure-time prefix does not exist, but is under an NFS mount, the compiler will cause automount traffic, NFS timeouts, etc. ... However, I think it's clear that the problems in (1) are more severe than the problems in (2), on several grounds: I actually think the problems with 1 (b) are artificial and should not be taken into account. 1(b) is a common and very real problem, especially on windows hosts. A hardcoded drive letter is pretty much guaranteed to be wrong. It makes the compiler unusable if that drive happens to be a removable drive or network share on the user's machine. Paul
Re: Searching configured and relocated prefix.
Andrew Pinski wrote: Yes, that's clever. But, you can create a symbolic link to the sysroot from each installation with a single command. And, your installer for third-party developers can do that for you. What are the equivalent to symbolic links on Windows and I am not talking about cygwin either? That's a good question. In Vista, the answer is symbolic links: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/symbolic_links.asp Before Vista, there's no solution short of cp. However, you still have the --sysroot command-line option. And, if you're worried about Windows, see Paul's response; the problems I've described are particularly bad on Windows, and the developer-base there is often less used to GNU software, so the problems are even weirder. -- Mark Mitchell CodeSourcery [EMAIL PROTECTED] (650) 331-3385 x713
Re: Searching configured and relocated prefix.
Before Vista, there's no solution short of cp. However, you still have the --sysroot command-line option. And, if you're worried about Windows, see Paul's response; the problems I've described are particularly bad on Windows, and the developer-base there is often less used to GNU software, so the problems are even weirder. But isn't Paul's response a confirmation that it is a bug in Windows for not having a stat cache and not knowing when a drive map becomes valid? If so I would have hoped that Codesourcery would have filed a bug with Microsoft already about how bad performance problem it is instead of now trying to work around in real already working code. Thanks, Andrew Pinski
Re: Searching configured and relocated prefix.
Andrew Pinski wrote: Before Vista, there's no solution short of cp. However, you still have the --sysroot command-line option. And, if you're worried about Windows, see Paul's response; the problems I've described are particularly bad on Windows, and the developer-base there is often less used to GNU software, so the problems are even weirder. But isn't Paul's response a confirmation that it is a bug in Windows for not having a stat cache and not knowing when a drive map becomes valid? If so I would have hoped that Codesourcery would have filed a bug with Microsoft already about how bad performance problem it is instead of now trying to work around in real already working code. Are you suggesting that we ship software that performs poorly on one of the most popular systems actually in the field because, in the abstract, those systems could be better? I would expect that most large software applications (for *all* operating systems) contain comments like: /* On some versions of the OS, we have to do X to workaround Y. */ It's just cutting off our nose to spite our face to ship software that doesn't work well and tell users wait until your system distributor fixes your OS. Even for most GNU/Linux users, that would be untenable; they're not system hackers, and they only get to upgrade when RHEL or SuSE or Debian or ... distributes new packages. -- Mark Mitchell CodeSourcery [EMAIL PROTECTED] (650) 331-3385 x713
Re: Searching configured and relocated prefix.
On Jul 23, 2006, at 10:44 AM, Ian Lance Taylor wrote: Me too. Except now you have suggested that we change the current behavior which you already suggested at the GCC summit we should not do at least not for a couple of release for warning people. You cannot have it both ways, people already depend on this behavior, I am one example. -- Pinski
Re: Searching configured and relocated prefix.
On Sun, Jul 23, 2006 at 11:40:38AM -0700, Andrew Pinski wrote: Before Vista, there's no solution short of cp. However, you still have the --sysroot command-line option. And, if you're worried about Windows, see Paul's response; the problems I've described are particularly bad on Windows, and the developer-base there is often less used to GNU software, so the problems are even weirder. But isn't Paul's response a confirmation that it is a bug in Windows for not having a stat cache and not knowing when a drive map becomes valid? If so I would have hoped that Codesourcery would have filed a bug with Microsoft already about how bad performance problem it is instead of now trying to work around in real already working code. This shows that you didn't really think about Paul's response. Please try again. The drive may be something else - like a floppy drive or other extremely slow media. This happens almost constantly. I think that in stage 1, we should switch to not searching any of the configured paths in favor of the relocated paths. Carlos has been working on patches for this. I'm sure it will break a few unexpected configurations. When it does, we can adjust the behavior as necessary, or document useful tricks for working with such things in the manual. -- Daniel Jacobowitz CodeSourcery
Re: Searching configured and relocated prefix.
On Jul 23, 2006, at 11:48 AM, Mark Mitchell wrote: Are you suggesting that we ship software that performs poorly on one of the most popular systems actually in the field because, in the abstract, those systems could be better? Maybe we just have to force the issue on people. As I mentioned to Ian, you are suggesting we change the behavior which people already depend on which is not something which anyone wants to do. In fact the current behavior is well defined and has a purpose in that you don't need to extra stuff to get a common set of other tools/sysroot for different compilers that would install at the same location. If we change it, we should be able to remove other options and features without going through a deprecation period either. We should not have a double standard here, just because it is a performance issue and other people are confused about the current behavior. I would expect that most large software applications (for *all* operating systems) contain comments like: /* On some versions of the OS, we have to do X to workaround Y. */ Yes but most of those because people don't think about filing bug reports. We need to file the bug reports with the OS. Also Daniel's point about a slow drive, there is again a reason why caches exist to speed up the accesses (which includes stats). If people don't file bugs, the work around stays forever and gets stuck there. In fact I think we have some work arounds in GCC which can be safely removed, like the comment size work around for Sun's c compiler. There are others like all the bool issues which we now work around via using char instead of _Bool. Thanks, Andrew Pinski
Re: Searching configured and relocated prefix.
Andrew Pinski wrote: On Jul 23, 2006, at 11:48 AM, Mark Mitchell wrote: Are you suggesting that we ship software that performs poorly on one of the most popular systems actually in the field because, in the abstract, those systems could be better? Maybe we just have to force the issue on people. One of the purposes of FSF GCC development is to get more people to use GCC. That's more likely to happen if we make GCC work well for them. We should not have a double standard here, just because it is a performance issue and other people are confused about the current behavior. We should not try so hard to be consistent that we do the wrong thing. It's good to have procedures and rules, but not to the point that they are absolutes. Decisions about these kinds of things are of necessity only semi-algorithmic. This isn't like a -f option which is a documented feature. This is the current behavior of the compiler, which I expect you'd find most people consider to be odd, despite the fact that there is some utility in the current approach. I think it would be wrong to make this change now (it's clearly neither Stage 3 nor regression-only material), but I see no reason not to make it in 4.2. /* On some versions of the OS, we have to do X to workaround Y. */ Yes but most of those because people don't think about filing bug reports. We need to file the bug reports with the OS. Sure, filing OS bug reports is good. It's just orthogonal; we still have to build software that works with the systems users are using. -- Mark Mitchell CodeSourcery [EMAIL PROTECTED] (650) 331-3385 x713
Re: Searching configured and relocated prefix.
On Sunday 23 July 2006 19:40, Andrew Pinski wrote: Before Vista, there's no solution short of cp. However, you still have the --sysroot command-line option. And, if you're worried about Windows, see Paul's response; the problems I've described are particularly bad on Windows, and the developer-base there is often less used to GNU software, so the problems are even weirder. But isn't Paul's response a confirmation that it is a bug in Windows for not having a stat cache and not knowing when a drive map becomes valid? It's not uncommon for the response times of absent/corrupt physical media or remote network devices to be many seconds. This delay typically occurs every time the device is accessed. This isn't a windows bug, it's an unavoidable property of that device. You argument seems to be that the current [undocumented] behavior is useful to me, so we shouldn't change it. My counter argument is that the current behavior is demonstrably broken in several fairly normal situations. Your problem can be worked around by symlinking or copying a few files. My problem requires major system changes (possibly including reinstallation and reconfiguration of the whole system) to workaround. Given that there doesn't seem to be a single solution that can satisfy both constraints I think causing some inconvenience to some users is vastly preferable to making the compiler unusable for others. Paul
RE: Searching configured and relocated prefix.
Mark Mitchell wrote: Sent: Monday, July 24, 2006 4:59 AM Andrew Pinski wrote: On Jul 14, 2006, at 1:17 PM, Carlos O'Donell wrote: We currently search both the relocated compilers prefix and the originally configured prefix. Should a relocated compiler be searching both directories? Yes because someone might have just relocated the compiler but not the rest of the tools. The main use of this is that untar a common sysroot and use different versions of the compiler which normally would be installed under that common location. There are benefits and costs in either direction. 1. If we search both locations (i.e., for a relocated compiler, search the configured-time prefix and the installation-time prefix), we get the following set of problems: (a) If the configure-time prefix exists (for unrelated reasons) the compiler finds files in the configure-time prefix, even though neither the system administrator or user expects that. 1(a) is reported quite reguarly on mingw32 user lists. I think I hit it myself when I first started comparing different versions of the compiler. This is a typical scenario on windows Configure-time prefix set to /mingw by binary release manager User installs in C:\mingw A new release is made available. Configure-time prefix is again /mingw, since the release manger just reused the old build script. User chooses to install in, say, C:\gcc-3.4.5-2\mingw, to keep thing from getting confused. The new compiler, run from C:, looks in /mingw first and gets it wrong. Confused user reports a bug. So my response to Should a relocated compiler be searching both directories on mingw32?, would be No, preferably only the relative-prefix. The --enable-win32-registry option, combined with an install-time regedit script, could still be used to force a common, absolute 'sysroot'. Danny
Re: Searching configured and relocated prefix.
Daniel Jacobowitz [EMAIL PROTECTED] writes: [...] | I think that in stage 1, we should switch to not searching any of the | configured paths in favor of the relocated paths. Carlos has been | working on patches for this. I'm sure it will break a few | unexpected configurations. When it does, we can adjust the behavior as | necessary, or document useful tricks for working with such things in | the manual. 100% agreed. -- Gaby
Re: Searching configured and relocated prefix.
Yes, that's clever. But, you can create a symbolic link to the sysroot from each installation with a single command. And, your installer for third-party developers can do that for you. Beware - the linker expands symlinks to determine if a library comes from the sysroot or not. It affects libraries which are really linker scripts (like libc.so); the linker will relocate the paths within *if* the library exists - truely exists - in the sysroot directory. Symlinks *to* the sysroot location work, symlinks *from* the sysroot location break the linker. At least, last time I was bitten by this, that's how it worked.
Re: Searching configured and relocated prefix.
On Jul 14, 2006, at 1:17 PM, Carlos O'Donell wrote: We currently search both the relocated compilers prefix and the originally configured prefix. Should a relocated compiler be searching both directories? Yes because someone might have just relocated the compiler but not the rest of the tools. The main use of this is that untar a common sysroot and use different versions of the compiler which normally would be installed under that common location. Thanks, Andrew Pinski
