Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
On 1/6/2011 4:25 PM, Joaquim Luis wrote: By the way, isn't there some new stuff about side by side installs or something -- that's supposed to help dll hell? I think you are referring to the manifests but here also MS has changed their mind (aleluia). VC2010 doesn't have them anymore which IMO is a compelling reason to use this compiler whenever possible. This is what I was talking about: http://msdn.microsoft.com/en-us/library/aa376307%28VS.85%29.aspx It does look like it is (or was) MS's solution to dll hell -- however, I remember it causing much pain for wxPython (particularly for py2exe), and I'm happy to defer to the experts here that seem to know that the cure was worse than the disease. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
2011/1/7 Jason Roberts jason.robe...@duke.edu Thanks for your thoughts. Based on them, I'd recommend the following two things be created. 1. GDAL windows installation program, or at minimum, a wiki page that says how to install the GDAL libraries and utilities (executables and Python scripts) to \Program Files\GDAL\... Perhaps a quick compromise would be for Tamas's build system to produce a .zip that would have everything in a suitable directory structure and for wiki page to instruct the user just unzip this to \Program Files\GDAL\ Jason, What would be the suitable directory structure? I'm keen to provide an installer which could place the files to the right location. By using WIXhttp://wix.sourceforge.net/it's could be generated automatically by the command line tools candle.exe and light.exe during the build process easily. However naming the install root folder to GDAL doesn't seem to be a right thing as we provide other packages like mapserver as well. For the sake of simplicitly I could imagine to place everything in a common folder (like SDKBuilds) and add a shortcut for invoking a command prompt (for starting the commandline tools) and a shortcut to uninstall the package. I would also mention OSGeo as the root, but I'm not sure how it will violate the files if a similar installer will ever derived from an OSGeo4W bundle. (We may probably warn the user not to install both versions side by side) I may also be mention that by default referring to the programsfolder in the installation process may select different folders depending on the architecture (Win32/Win64) or the OS version. I don't think it would be a good way to force everything to be in C:\Program Files which contains the x64 binaries on x64 platforms for example or it may reside in various logical drives on a particular system. It would probably be reasonable to use something like SHGetSpecialFolderLocationhttp://msdn.microsoft.com/en-us/library/bb762203%28v=vs.85%29.aspxto retrieve the current location in the loader program. 2. GDAL Python bindings installation program. This could be easily created using the standard Python distutils stuff I've been mentioning, and would install the bindings to the normal Python place. The bindings would ideally be modified to check for and explicitly load libraries from \Program Files\GDAL\... This would eliminate the need to modify the PATH variable. That's ok with me as a separate installer provided by distutils. Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Frank, I would suggest building it as an installer .exe, perhaps using NSIS as I did for FWTools, or perhaps the method mentioned by Jurgen produces a nice installer. If possible, I would suggest the final output be a .msi (Windows Installer package) rather than a .exe. In an effort to make the average user more secure, Microsoft continues to clamp down on the flexibility afforded to .exe installers, due to the trouble of disambiguating them from malicious programs. It was a big shock when Windows Vista was introduced and it prevented .exe installers from truly running with administrative rights even if the logged on user was an administrator. Everybody knows about that issue now and has learned how to modify their .exe to request those rights (causing the user to be prompted for confirmation) but who knows what Microsoft will do next. For that reason, I believe it would be better to bite the bullet and adhere to their recommended installation format. Microsoft's Visual Studio has long had an easy mechanism for building .msi installers. It is what I would recommend for this, unless you prefer to use a FOSS alternative, in which case I cannot advise on what would be good. There are many. Perhaps Jürgen has a better idea. One question not discussed is whether GDAL should be minimalist or maximalist. That is, do we want to include as many formats as possible despite the fact that it drags in lots of supporting libraries? If we just use whatever decision OSGeo4W uses then we will have a fairly maximalist solution which is ok I suppose but might make integration of the resulting GDAL in other complex applications messy. IMO, there are two primary goals of this packaging of GDAL: provide easy access to just the GDAL command line utilities (separate from larger tool suites such as FWTools and OSGeo4W), and provide the GDAL libraries in a well-known location so they can be located by the various GDAL bindings. I think a secondary goal is to provide GDAL libraries in a well-known location for integration by arbitrary applications. Because the GDAL binaries will not normally be in the PATH, only when the user is wanting to use the GDAL utilities, the mere presence of a large number of supporting libraries will not be an issue. As you noted, it is when it comes to integration that there is a problem. I don't think there's an easy answer for the minimalist/maximalist question. I think it would be best to err on the maximimalist side, with the idea of providing as fully-functional command line suite as possible. But if there are particular libraries that are known to cause problems for popular integration scenarios but are rarely used by command-line users, then leave them out. I don't know how to identify such libraries; the GDAL team and power users are the best source of that info. Over time, you can hopefully ease any problems by moving things to plugins that are only loaded on demand, allowing someone to integrate without hitting a problem unless the plugin is loaded. In the end, integrators are likely to be skilled programmers and their fallback position will be to build a private copy of GDAL with the select dependencies they need. I would like to suggest production of such an installer be done in such a way that the generating scripts live in SVN somewhere, and with instructions for the process in the wiki so it isn't all tied to one person (as FWTools was). Man, I'm really tempted to leap into this myself but I have so many other outstanding items right now. I am willing to assist in an effort by a small (2-3 people?) team. I would be happy offer opinions, review, and test but am not available to lead at this point. As the developer of the primary source of Windows builds of GDAL, I think Tamas is the logical leader for this if he is available. Thank you for seeing this as an important priority. Jason ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Tamas, WIX looks like a great technology for building the installation package. I've never used it but I took a quick look and it seems to provide what is needed. As I understand it, you are pondering whether it would be better to have GDAL in Program Files\OSGeo\GDAL or in Program Files\GDAL. Is it simply a question aesthetics or principle, in which it seems proper to put all OSgeo projects under Program Files\OSGeo, but there could be problems coordinating the efforts of multiple projects to adhere to that carefully and not mash each other's files? If that summarizes the issue, then I'd recommend going with the more practical, less principled approach: put GDAL in Program Files\GDAL, and OSGeo4W in Program Files\OSGeo4W. It could get messy if both installers have to be able to create the Program Files\OSGeo directory but not necessarily delete it when they are uninstalled, because there could be another OSGeo project in there. Along those lines, I would suggest that if GDAL plans to support side-by-side installations of GDAL itself, then we need to contemplate carefully whether to use Program Files\GDAL\GDAL-X.Y or Program Files\GDAL-X.Y. I think either one would be ok, so long as whoever writes the installer thinks out all the side-by-side installation/uninstallation scenarios. Another question, also already raised, is whether to have just two or all three version numbers in the installation directory. I think that question depends on whether you ever need to support bugfix releases installed side-by-side (e.g. 1.8.0 and 1.8.1), and also whether the bindings and other integrators will need to bind to specific bugfix releases or not. For example, if you guarantee that all 1.8.x releases will have compatible ABIs-that you will never introduce a change that will break the binary interface without increasing the minor build number-then it would be ok to just use X.Y in the directory name. But if that cannot be guaranteed, then you need to support X.Y.Z so that bindings and other integrators can locate the specific bugfix release that they are compatible with. Following up on that point: I think we've agreed that we do not want the user to have to set the PATH in order to have the GDAL bindings work. Therefore the bindings must be changed to look for the GDAL libraries in the well-known location. Because they currently use implicit linking, they are tightly coupled to a particular version of the libraries. The user will have to be aware that they must install a compatible pair of bindings + GDAL. The GDAL team should decide right now about the X.Y vs. X.Y.Z compatibility question. Regarding x86 vs. x64. The GDAL installer should follow the standard Microsoft convention. On an x64 machine, the x64 GDAL utils/libs should go in \Program Files and the x86 utils/libs should go in \Program Files (x86). The bindings and other integrators must be aware of x86 vs. x64 and locate libs from the correct directory. I agree that calling SHGetSpecialFolderLocation with an appropriate CSIDL is an appropriate way to find the Program Files directory (addressing issue noted by Joaquim and others that Program Files is localized). It is probably ok to use the ProgramFiles and ProgramFiles(x86) environment variables if calling the Win32 API is not easy for particular bindings. Jason From: Tamas Szekeres [mailto:szeker...@gmail.com] Sent: Friday, January 07, 2011 5:26 AM To: Jason Roberts Cc: Frank Warmerdam; gdal-dev@lists.osgeo.org Subject: Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0 2011/1/7 Jason Roberts jason.robe...@duke.edu Thanks for your thoughts. Based on them, I'd recommend the following two things be created. 1. GDAL windows installation program, or at minimum, a wiki page that says how to install the GDAL libraries and utilities (executables and Python scripts) to \Program Files\GDAL\... Perhaps a quick compromise would be for Tamas's build system to produce a .zip that would have everything in a suitable directory structure and for wiki page to instruct the user just unzip this to \Program Files\GDAL\ Jason, What would be the suitable directory structure? I'm keen to provide an installer which could place the files to the right location. By using WIX http://wix.sourceforge.net/ it's could be generated automatically by the command line tools candle.exe and light.exe during the build process easily. However naming the install root folder to GDAL doesn't seem to be a right thing as we provide other packages like mapserver as well. For the sake of simplicitly I could imagine to place everything in a common folder (like SDKBuilds) and add a shortcut for invoking a command prompt (for starting the commandline tools) and a shortcut to uninstall the package. I would also mention OSGeo as the root, but I'm not sure how it will violate the files if a similar installer will ever derived from an OSGeo4W bundle. (We may probably warn the user not to install both versions
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
On 07-01-2011 15:07, Jason Roberts wrote: Tamas, WIX looks like a great technology for building the installation package. I've never used it but I took a quick look and it seems to provide what is needed. As I understand it, you are pondering whether it would be better to have GDAL in Program Files\OSGeo\GDAL or in Program Files\GDAL. Is it simply a question aesthetics or principle, in which it seems proper to put all OSgeo projects under Program Files\OSGeo, but there could be problems coordinating the efforts of multiple projects to adhere to that carefully and not mash each other's files? If that summarizes the issue, then I'd recommend going with the more practical, less principled approach: put GDAL in Program Files\GDAL, and OSGeo4W in Program Files\OSGeo4W. It could get messy if both installers have to be able to create the Program Files\OSGeo directory but not necessarily delete it when they are uninstalled, because there could be another OSGeo project in there. Let me just recall that OSGeo4W proposes the default install directory as C:\OSGeo4W This is just my opinion and I feel quite at rest because I'm able to build GDAL and install it wherever I wont, but I would really hate that an installer would force me to install in a location contrary to my will. In particular when that location has blanks in the name (Program Files). But, as I said it's just my private view. Joaquim ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
On 11-01-07 10:07 AM, Jason Roberts wrote: put GDAL in Program Files\GDAL, and OSGeo4W in Program Files\OSGeo4W. Jason, I would note that OSGeo4W installs in C:\OSGeo4W by default, and there are no plans currently to change this. OSGeo4W will continue to use it's internal copy of GDAL. It could get messy if both installers have to be able to create the Program Files\OSGeo directory but not necessarily delete it when they are uninstalled, because there could be another OSGeo project in there. I would prefer to stick to C:\Program Files\GDAL. Along those lines, I would suggest that if GDAL plans to support side-by-side installations of GDAL itself, then we need to contemplate carefully whether to use Program Files\GDAL\GDAL-X.Y or Program Files\GDAL-X.Y. I think either one would be ok, so long as whoever writes the installer thinks out all the side-by-side installation/uninstallation scenarios. I prefer to just use the major.minor version as the subdirectory name. C:\Program Files\GDAL\1.7\ Another question, also already raised, is whether to have just two or all three version numbers in the installation directory. I think that question depends on whether you ever need to support bugfix releases installed side-by-side (e.g. 1.8.0 and 1.8.1), and also whether the bindings and other integrators will need to bind to specific bugfix releases or not. For example, if you guarantee that all 1.8.x releases will have compatible ABIs—that you will never introduce a change that will break the binary interface without increasing the minor build number—then it would be ok to just use X.Y in the directory name. But if that cannot be guaranteed, then you need to support X.Y.Z so that bindings and other integrators can locate the specific bugfix release that they are compatible with. It is an explicit guarantee that point releases do not change the C *or* C++ public ABI. It is an explicit intention that a point release can be installed over a previous release without impacting any applications using GDAL. This extends to the point that plugins should be compatible across point releases. Following up on that point: I think we’ve agreed that we do not want the user to have to set the PATH in order to have the GDAL bindings work. I do *not* agree that installing GDAL would necessarily insert it into the global PATH. It might, or might not, but I'm still very concerned about possible interference with other applications using GDAL. I'm not saying I will never agree to it, but I remain very cautious about interference in the global space. Particularly since we are likely to carry along lots of other DLLs (zlib, xerces, etc). If this installer is to be a product of the project, I would suggest you write up a proposal as an RFC and we work from that. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Programmer for Rent ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Joaquim, I hear your pain about having the space in the name. It would probably not be too much trouble for the installer to allow the user to change the installation directory. That is a basic option of most installers so it would probably not be hard to implement with a WIX-built installer. But I should reiterate that a requirement of this package is that the GDAL bindings (installed as separate packages) must be able to locate the GDAL libraries. The one approach mentioned so far is to put the binaries in a well-known location. If the bindings were designed to look for the libraries in a well-known location, they would be broken if the user installed them to a different location. Therefore it is important to pick a default location and recommend that only very knowledgeable users select a different place. My view is that adhering to Microsoft's best practice of using \Program Files outweighs the convenience of using a directory off a root partition that does not have a space in the name. An alternative idea is to store the location of the GDAL libraries in the registry and then permitting them to be installed anywhere with no consequences. The bindings would read the registry to determine their location. This is also a fairly standard way to do things on Windows, but it would be slightly more complicated for the bindings. Jason From: Joaquim Luis [mailto:jl...@ualg.pt] Sent: Friday, January 07, 2011 10:27 AM To: Jason Roberts Cc: 'Tamas Szekeres'; gdal-dev@lists.osgeo.org Subject: Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0 On 07-01-2011 15:07, Jason Roberts wrote: Tamas, WIX looks like a great technology for building the installation package. I've never used it but I took a quick look and it seems to provide what is needed. As I understand it, you are pondering whether it would be better to have GDAL in Program Files\OSGeo\GDAL or in Program Files\GDAL. Is it simply a question aesthetics or principle, in which it seems proper to put all OSgeo projects under Program Files\OSGeo, but there could be problems coordinating the efforts of multiple projects to adhere to that carefully and not mash each other's files? If that summarizes the issue, then I'd recommend going with the more practical, less principled approach: put GDAL in Program Files\GDAL, and OSGeo4W in Program Files\OSGeo4W. It could get messy if both installers have to be able to create the Program Files\OSGeo directory but not necessarily delete it when they are uninstalled, because there could be another OSGeo project in there. Let me just recall that OSGeo4W proposes the default install directory as C:\OSGeo4W This is just my opinion and I feel quite at rest because I'm able to build GDAL and install it wherever I wont, but I would really hate that an installer would force me to install in a location contrary to my will. In particular when that location has blanks in the name (Program Files). But, as I said it's just my private view. Joaquim ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
2011/1/7 Jason Roberts jason.robe...@duke.edu WIX looks like a great technology for building the installation package. I’ve never used it but I took a quick look and it seems to provide what is needed. Jason, I've already used WIX many times and I'm very satisfied with it. As I understand it, you are pondering whether it would be better to have GDAL in Program Files\OSGeo\GDAL or in Program Files\GDAL. Is it simply a question aesthetics or principle, in which it seems proper to put all OSgeo projects under Program Files\OSGeo, but there could be problems coordinating the efforts of multiple projects to adhere to that carefully and not mash each other’s files? If that summarizes the issue, then I’d recommend going with the more practical, less principled approach: put GDAL in Program Files\GDAL, and OSGeo4W in Program Files\OSGeo4W. It could get messy if both installers have to be able to create the Program Files\OSGeo directory but not necessarily delete it when they are uninstalled, because there could be another OSGeo project in there. Along those lines, I would suggest that if GDAL plans to support side-by-side installations of GDAL itself, then we need to contemplate carefully whether to use Program Files\GDAL\GDAL-X.Y or Program Files\GDAL-X.Y. I think either one would be ok, so long as whoever writes the installer thinks out all the side-by-side installation/uninstallation scenarios. Just by thinking about the details if we implement a smart dll loader which could in fact work with any locations to load the dll-s why do we require to use a specific folder as the install location? The installer may create a registry entry or a config file in a specific place where the necessary dependencies are installed actually. The loader would be responsible to query for a compatible entry and use that location when loading the dependent dll-s. If no entries found the loader would provide an error that a dependent setup should also be executed. Another question, also already raised, is whether to have just two or all three version numbers in the installation directory. I think that question depends on whether you ever need to support bugfix releases installed side-by-side (e.g. 1.8.0 and 1.8.1), and also whether the bindings and other integrators will need to bind to specific bugfix releases or not. For example, if you guarantee that all 1.8.x releases will have compatible ABIs—that you will never introduce a change that will break the binary interface without increasing the minor build number—then it would be ok to just use X.Y in the directory name. But if that cannot be guaranteed, then you need to support X.Y.Z so that bindings and other integrators can locate the specific bugfix release that they are compatible with. It would not be a problem in the scope of my comments above. The user could install multiple versions anywhere and the loader would be responsible to find out the corresponding version at run time. Regarding x86 vs. x64. The GDAL installer should follow the standard Microsoft convention. On an x64 machine, the x64 GDAL utils/libs should go in \Program Files and the x86 utils/libs should go in \Program Files (x86). The bindings and other integrators must be aware of x86 vs. x64 and locate libs from the correct directory. I agree that calling SHGetSpecialFolderLocation with an appropriate CSIDL is an appropriate way to find the Program Files directory (addressing issue noted by Joaquim and others that Program Files is localized). It is probably ok to use the ProgramFiles and ProgramFiles(x86) environment variables if calling the Win32 API is not easy for particular bindings. We could probably implement the loader in C an use SWIG to map the functions to the various languages. The implementation would be the same for all languages in this regard. The only issue here is that where to place the loader.dll (providing the core implementation) which would also rely on the Windows dll search rules. For python you could easily say to create a pyd installed along with the python files, but in may not be so trivial for the other languages. I'm tending to think that this part should probably be implemented in a generic way (which doesn't change frequently) and be installed in a common folder available in the PATH. Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Frank, Thanks for the clarification of the various issues. I will start an RFC for this next week. Jason -Original Message- From: Frank Warmerdam [mailto:warmer...@pobox.com] Sent: Friday, January 07, 2011 10:29 AM To: Jason Roberts Cc: gdal-dev@lists.osgeo.org Subject: Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0 On 11-01-07 10:07 AM, Jason Roberts wrote: put GDAL in Program Files\GDAL, and OSGeo4W in Program Files\OSGeo4W. Jason, I would note that OSGeo4W installs in C:\OSGeo4W by default, and there are no plans currently to change this. OSGeo4W will continue to use it's internal copy of GDAL. It could get messy if both installers have to be able to create the Program Files\OSGeo directory but not necessarily delete it when they are uninstalled, because there could be another OSGeo project in there. I would prefer to stick to C:\Program Files\GDAL. Along those lines, I would suggest that if GDAL plans to support side-by-side installations of GDAL itself, then we need to contemplate carefully whether to use Program Files\GDAL\GDAL-X.Y or Program Files\GDAL-X.Y. I think either one would be ok, so long as whoever writes the installer thinks out all the side-by-side installation/uninstallation scenarios. I prefer to just use the major.minor version as the subdirectory name. C:\Program Files\GDAL\1.7\ Another question, also already raised, is whether to have just two or all three version numbers in the installation directory. I think that question depends on whether you ever need to support bugfix releases installed side-by-side (e.g. 1.8.0 and 1.8.1), and also whether the bindings and other integrators will need to bind to specific bugfix releases or not. For example, if you guarantee that all 1.8.x releases will have compatible ABIs-that you will never introduce a change that will break the binary interface without increasing the minor build number-then it would be ok to just use X.Y in the directory name. But if that cannot be guaranteed, then you need to support X.Y.Z so that bindings and other integrators can locate the specific bugfix release that they are compatible with. It is an explicit guarantee that point releases do not change the C *or* C++ public ABI. It is an explicit intention that a point release can be installed over a previous release without impacting any applications using GDAL. This extends to the point that plugins should be compatible across point releases. Following up on that point: I think we've agreed that we do not want the user to have to set the PATH in order to have the GDAL bindings work. I do *not* agree that installing GDAL would necessarily insert it into the global PATH. It might, or might not, but I'm still very concerned about possible interference with other applications using GDAL. I'm not saying I will never agree to it, but I remain very cautious about interference in the global space. Particularly since we are likely to carry along lots of other DLLs (zlib, xerces, etc). If this installer is to be a product of the project, I would suggest you write up a proposal as an RFC and we work from that. Best regards, -- ---+ -- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Programmer for Rent ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
On 1/6/11 7:55 PM, Frank Warmerdam wrote: One question not discussed is whether GDAL should be minimalist or maximalist. That is, do we want to include as many formats as possible despite the fact that it drags in lots of supporting libraries? +1 for Maximalist. We want as many things as possible to just work If we just use whatever decision OSGeo4W uses then we will have a fairly maximalist solution which is ok I suppose but might make integration of the resulting GDAL in other complex applications messy. People that are integrating into complex apps will likely roll their own anyway. I would like to suggest production of such an installer be done in such a way that the generating scripts live in SVN somewhere, and with instructions for the process in the wiki so it isn't all tied to one person (as FWTools was). Absolutely! On 1/6/11 8:23 PM, Jürgen E. Fischer wrote: You get a desktop link and a start menu entry that both opens a command line window from where you can use GDAL very nice. and start python with gdal available. what Python gets started? Can you start Python from any other command line and have it work, or do you need ENV VARS set up? On 1/7/11 2:26 AM, Tamas Szekeres wrote: However naming the install root folder to GDAL doesn't seem to be a right thing as we provide other packages like mapserver as well. well, either you can have GDAL in GDAL, and mapserver in MapServer. or... I would also mention OSGeo as the root, If there are a number of things under the OSGeo umbrella provided by the same people, then OSGeo is a fine root. Note that I'd have: Program Files\OSGeo\GDAL Program Files\OSGeo\Mapserver rather than dumping everything directly in OSGeo\ -- though maybe a shared bin does make sense? Oh for some standards! but I'm not sure how it will violate the files if a similar installer will ever derived from an OSGeo4W bundle. (We may probably warn the user not to install both versions side by side) Ideally, the OSGEo4W bundle will be compatible -- that really would be better --i.e. a GDAL binary is a subset of an OSGeo install. But if that's not possible, then we'll just have to use a different name than OSGeo4W does. IMO, there are two primary goals of this packaging of GDAL: provide easy access to just the GDAL command line utilities (separate from larger tool suites such as FWTools and OSGeo4W), and provide the GDAL libraries in a well-known location so they can be located by the various GDAL bindings. I think a secondary goal is to provide GDAL libraries in a well-known location for integration by arbitrary applications. good summary. For example, if you guarantee that all 1.8.x releases will have compatible ABIs—that you will never introduce a change that will break the binary interface guarantee is a strong word, but that is the idea as I understand it. Therefore the bindings must be changed to look for the GDAL libraries in the well-known location. (or more than one, but I think one is a good start). It seems on *nix, for instance, this would generally happen in the build stage (actually configure) -- the builder specifies where they want it installed, and everything is built to match that. That's a fine model to follow. I don't know if there is any equivalent of ./configure for Windows building, but a simple script could at least do this bit easily enough. The user will have to be aware that they must install a compatible pair of bindings + GDAL. The GDAL team should decide right now about the X.Y vs. X.Y.Z compatibility question. yes -- and If we go with X.Y, then you should be able to drop an updated point release of GDAL in, without updating the bindings. Is that good, however, or are the bindings also updated, and tightly related? I think you could do X.Y, but have the bindings check for point version compatibility, so you'd get a Warning if you were using a mismatched binding. Let me just recall that OSGeo4W proposes the default install directory as C:\OSGeo4W not a bad option either, but is that difficult/impossible/ugly with recent MS security policies (still on XP here...)? Ugly though it is, there is a lot to be said for conforming to MS recommended practice. Following up on that point: I think we’ve agreed that we do not want the user to have to set the PATH in order to have the GDAL bindings work. I do *not* agree that installing GDAL would necessarily insert it into the global PATH. I think we have ways of getting the bindings to work without global PATH. It might, or might not, but I'm still very concerned about possible interference with other applications using GDAL. I'd want global PATH settings for command line tool usage. using it for finding dlls seems like a really bad idea. With that in mind, perhaps the command line tools should be in a different dir from the dlls, so that you can put the former on your global PATH without breaking any linking. Whether the
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Hi Chris, On Fri, 07. Jan 2011 at 12:03:47 -0800, Christopher Barker wrote: On 1/6/11 8:23 PM, Jürgen E. Fischer wrote: You get a desktop link and a start menu entry that both opens a command line window from where you can use GDAL very nice. and start python with gdal available. what Python gets started? Can you start Python from any other command line and have it work, or do you need ENV VARS set up? The python shipped with the installer (just like in OSGeo4W). On 1/7/11 2:26 AM, Tamas Szekeres wrote: IMO, there are two primary goals of this packaging of GDAL: provide easy access to just the GDAL command line utilities (separate from larger tool suites such as FWTools and OSGeo4W), and provide the GDAL libraries in a well-known location so they can be located by the various GDAL bindings. I think a secondary goal is to provide GDAL libraries in a well-known location for integration by arbitrary applications. good summary. hm, just an idea - creatensis.pl could even be generized to pick an arbitrary OSGeo4W package and produce a NSIS installer with all the necessary dependencies (even as CGI - I created the package on Linux). If the GDAL package in OSGeo4W did provide a desktop and start link and start menu link the nsi wouldn't need to create those and the extra batch file could be dropped. So the current customization wouldn't be needed anymore. That also would work for meta package (eg. something like gdal-min and gdal-max). Jürgen -- Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-20 Dipl.-Inf. (FH) Rheinstraße 13Fax. +49-4931-918175-50 Software Engineer D-26506 Norden http://www.norbit.de -- norBIT Gesellschaft fuer Unternehmensberatung und Informationssysteme mbH Rheinstrasse 13, 26506 Norden GF: Jelto Buurman, HR: Amtsgericht Emden, HRB 5502 ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
On 01/06/2011 01:18 AM, Tamas Szekeres wrote: Chris, Good points below, but having the compiled gdal binaries (and the binaries of the dependent libraries) in hand, which is the right way to install those files on Windows? (Assuming we don't provide python.exe and the related files in the package)? I mean which install actions should be done in detail? 2011/1/5 Christopher Barker: Windows binaries built in MinGW are available at: http://map.hut.fi/files/Geoinformatica/win32/ The Geoinformatica-yy-mm-dd.zip contains GDAL (usually a development version), Perl-GDAL, Perl, and many other things. good for MinGW users, I suppose -- I remember them not working for me, tough I can't recall how or why not. They also suffer from perhaps trying to be too much (though if it all worked, I wouldn't care, I have a fast network and large hard drive) I found it impossible to deliver a Windows binary solution, which would depend on the three main packages (Perl, GTK+, and GDAL) to be retrieved as binaries from some standard location - that's why I bundle them all together into a quite large (32MB) package. When I started there was only ActivePerl for Windows that was usable, but it turned out impossible for me to use because they use MS compilers. Now there is Strawberry Perl, which I might try again (or pay someone to try). It is good because it uses MinGW. There are binary installers for GTK+ for Windows, which install them into a shared location - but there are problems still and some pieces are missing that I need (Glade for example). The official GTK+ site has a long list of zips containing binaries. Despite the shared nature of GTK+, many projects include own binaries of them in Windows due to version mismatches etc. GDAL is available but again typically as MS compiler builds - which should not be a problem in theory because the bindings use it through the C API. I've tried to use those a couple of times without luck (compiling the bindings in MinGW was the problem). Maybe I should try again using binaries from Tamas' site. I agree that there could be a one main site for GDAL Windows binaries (something like http://www.gtk.org/download-windows.html). Tamas' site looks good but I'd like to have dev packages also (the SDK packages there look old) - just the header files should be enough. Then there could also be a more end-user type Windows installation package project for GDAL, which installs the stuff into some standard location (c:\Program Files\share?). For GTK+ that's provided by some folks at http://sourceforge.net/projects/gtk-win/ It would be neat to be able to just put to the Geoinformatica Windows installer subinstallations, which go and get Perl, GTK+ and GDAL for the user if she does not yet have them or has too old ones. In fact Geoinformatica should be simply installable through Perl's cpan application - but that does not even work in Linux yet :( (the reason is that the GDAL Perl module installation script is not intelligent enough and the version of the module in CPAN is horribly old and I really suck at managing that) Best regards, Ari ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
2011/1/6 Ari Jolma ari.jo...@gmail.com GDAL is available but again typically as MS compiler builds - which should not be a problem in theory because the bindings use it through the C API. I've tried to use those a couple of times without luck (compiling the bindings in MinGW was the problem). Maybe I should try again using binaries from Tamas' site. I agree that there could be a one main site for GDAL Windows binaries (something like http://www.gtk.org/download-windows.html). Tamas' site looks good but I'd like to have dev packages also (the SDK packages there look old) - just the header files should be enough. Ari, The SDK packages from http://vbkto.dyndns.org/sdk/ are exactly the same which have been used to compile the daily builds, so it should be up to date. The only thing may have to be done is to download the required version of the gdal sources in the root folder, because not all of the versions included in the package in order to keep the size as small as possible. BTW: What is the desired practice to install the gdal files + bindings along with a pre-installed perl runtime on Windows? Something like we have been discussing for python in this thread, do we have some desired install locations, environment settings or packaging conventions? Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Three months ago I put an announcement on the MapServer-Users list about a shell script I wrote to compile the whole MapServer-GDAL-PostGIS-PLR suite, with and without PHP/Python, in a separate user directory using only two commands. Separate installations can coexist in separate directory trees and all parameters of all programs can be adapted for eacht installation. Apache, PHP, PostGIS and Python are installed for each installation separately. The binaries can be transported to computers with the same system libraries. Since then,I tried it out on dozens of Virtual Machines in a Cloud virtual environment on Suse, Fedora and (prefered) Debian virtual machines, and it works for me without glitches. It's something like a low-level replacement for FWTools (actually, I wrote it when I heard from Frank that FWTools wasn't supported at the moment). It's Linux-only, although porting it to Windows should be possible with a msys/cygwin shell, even for native compiles. Since then I got no (zero) replies, so I don't know whether it doesn't work, isn't interesting, or people just didn't look at it. If it's worth your time and answers some of the questions from this discussion, please have a look and let me know what you think of it. And that includes doesn't work, not interesting, not adaptable enough, downright stupid. Here is the MS-Users posting: http://lists.osgeo.org/pipermail/mapserver-users/2010-October/066898.html Cheers, Jan On 01/05/11 23:32, Frank Warmerdam wrote: Folks, Wow, what a lot of discussion. From my perspective, I'm pleased with Tamas' provided binaries as a source for developers who want a GDAL SDK for a particular compiler version and choice of win32/win64. I like Jurgen's idea of building a stock GDAL binaries package with an installer (similar to FWTools) from OSGeo4W packages. Like some others, I'm also confused by the correct way of providing python binaries. I'm inclined to provide an included python in a self installer similar to FWTools, but I'd be pleased if the hardcore Pythonistas can get the magic ways of producing python binaries for any Python version working. Is there someone that wants to take on producing official GDAL binaries from OSGeo4W with a self installer sort of similar to FWTools, likely using the mechanism Jurgen described? I'm interested, but interest hasn't turned into action over quite a few months of interest. Best regards, ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Jason, I appreciate the expertise for all of you along with this thread, I could already gather quite some useful information from here for this reason. I must mention that my programming practice in Python can be considered as zero, this is the main reason that my issues may have trivial solutions for the hardcore pythonists but not trivial to me. Apologies for this inconvenience :-) Getting back to the original topic, you mention that the gdal binaries should be installed somewhere an set PATH, GDAL_DATA, PROJ_LIB and GDAL_DRIVER_PATH as a systemwide setting. This is where the problems of mine are starting. Modifying the PATH globally is a bad practice in 99% of the cases. The only case I'm aware of which may not be a problem when we make sure that only one version of such files (dll-s and executables) will ever be installed to a particular system. But this is not the case with the gdal binaries as I would expect at least a development or a stable version (and their x86/x64 variants) to coexist which should be used by the same user. The same problem may arise when we would like to install multiple versions to the site packages directory, how the versions of the files are maintained by the python runtime? In this regard I could mention something like what have been done with the .NET framework with the multiple versions of the packages installed simultaneously in the global assembly cache. The individual packages may specify the required minimum version of the referred packages loaded by the .NET runtime. Is this something that can be done with the python environment as well? (As opposed to this, the dumb solution of having a starting script to open a command prompt (and setting PYTHONPATH properly) would ensure multiple versions to be used at the same time, since those settings are applyed to the cmd process solely.) Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
On 01/06/2011 01:38 PM, Tamas Szekeres wrote: 2011/1/6 Ari Jolma ari.jo...@gmail.com mailto:ari.jo...@gmail.com GDAL is available but again typically as MS compiler builds - which should not be a problem in theory because the bindings use it through the C API. I've tried to use those a couple of times without luck (compiling the bindings in MinGW was the problem). Maybe I should try again using binaries from Tamas' site. I agree that there could be a one main site for GDAL Windows binaries (something like http://www.gtk.org/download-windows.html). Tamas' site looks good but I'd like to have dev packages also (the SDK packages there look old) - just the header files should be enough. Ari, The SDK packages from http://vbkto.dyndns.org/sdk/ are exactly the same which have been used to compile the daily builds, so it should be up to date. The only thing may have to be done is to download the required version of the gdal sources in the root folder, because not all of the versions included in the package in order to keep the size as small as possible. By the age I meant that the SDK packages are old releases (from 1310 to 1600 and not trunk for example - do I understand the release names correctly?) BTW: What is the desired practice to install the gdal files + bindings along with a pre-installed perl runtime on Windows? Something like we have been discussing for python in this thread, do we have some desired install locations, environment settings or packaging conventions? CPAN has only sources, thus cpan application which is the standard to download and install perl modules expects you to have a compiler. ActivePerl (in fact ActiveState, the company) maintains a repository of perl modules in binary versions, from where they can be simply downloaded and installed with another program. ActivePerl has tools for developing those binary packages. That's very similar to what Python has. I think ActiveState maintains its repository by itself - so if I just make the CPAN module intelligent enough it may end up there eventually. I think my Geo::Shapelib module was/is there. I think it would not make sense to include GDAL into such a binary perl module package. Thus GDAL would need to be separately installable - the module installer could probably be made to offer install it for the user if it existed somewhere. In short /me thinks the requirements are: 1) /me develop the perl bindings configure make procedure better 2) we make GDAL-dev.msi available at an URL. Ari Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
2011/1/6 Ari Jolma ari.jo...@gmail.com By the age I meant that the SDK packages are old releases (from 1310 to 1600 and not trunk for example - do I understand the release names correctly?) Ari, Those numbers are MSVC compiler numbers (according to http://trac.osgeo.org/gdal/browser/trunk/gdal/nmake.opt) not gdal version numbers. CPAN has only sources, thus cpan application which is the standard to download and install perl modules expects you to have a compiler. ActivePerl (in fact ActiveState, the company) maintains a repository of perl modules in binary versions, from where they can be simply downloaded and installed with another program. ActivePerl has tools for developing those binary packages. That's very similar to what Python has. I think ActiveState maintains its repository by itself - so if I just make the CPAN module intelligent enough it may end up there eventually. I think my Geo::Shapelib module was/is there. I think it would not make sense to include GDAL into such a binary perl module package. Thus GDAL would need to be separately installable - the module installer could probably be made to offer install it for the user if it existed somewhere. This is also the case with python and the other bindings: the gdal dll-s and dependencies should also be installed somewhere. Assuming we install the gdal-perl modules in a standard location (not sure where it is), do we have a common mechanism in the perl runtime to find the dependent dlls without having to violate system wide settings (like the PATH environment variable)? Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
On 01/06/2011 03:25 PM, Tamas Szekeres wrote: Assuming we install the gdal-perl modules in a standard location (not sure where it is), do we have a common mechanism in the perl runtime to find the dependent dlls without having to violate system wide settings (like the PATH environment variable)? :) I'm not completely satisfied (I'm also not sure I really understand the whole f* issue) with how Perl searches for dll's - especially dll import libs - in Windows. But that's configure/compile time issue. In compile time a Perl GDAL dll (for the bindings) is created and put into some location, which is known to Perl and Perl knows how to find it in runtime. Assuming I've built the binaries, then in runtime (when the GDAL module is called for) I think somebody (because the Perl GDAL module dll has a link to GDAL dll that was put there in compile time) asks for the GDAL dll's and they simply need to be available. Thus, if there is GDAL in the system (for example in c:\Program Files\Share\GDAL\bin) then that path needs to be in the PATH - I don't think there is a way around that. The PATH may be a temporary PATH set by somebody for Perl programs, but I think the policy should be that when you run GDAL.msi, the system default PATH is modified to have the path to the GDAL binaries. Best regards, Ari ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Ari The only wokaround which is satisfactory to me is to set the required enviroment at run time before the gdal bindings are used. In csharp we can do something like: string path = Environment.GetEnvironmentVariable(PATH); if (!path.Contains(gdal_bin_path)) { Environment.SetEnvironmentVariable(PATH, gdal_bin_path + ; + path); } Gdal.SetConfigOption(GDAL_DRIVER_PATH, gdal_driver_path); But it requires that loading the gdal dll is happening when the interface is used actually (in Gdal.SetConfigOption with the code above). This is the case with the .NET runtime, but I'm not sure how do we stand with the other bindings in this regard. Futher problem with this solution is that this startup code should be placed in the user's script and not sure how this could be implemented in the gdal bindings itself. Best regards, Tamas 2011/1/6 Ari Jolma ari.jo...@gmail.com On 01/06/2011 03:25 PM, Tamas Szekeres wrote: Assuming we install the gdal-perl modules in a standard location (not sure where it is), do we have a common mechanism in the perl runtime to find the dependent dlls without having to violate system wide settings (like the PATH environment variable)? :) I'm not completely satisfied (I'm also not sure I really understand the whole f* issue) with how Perl searches for dll's - especially dll import libs - in Windows. But that's configure/compile time issue. In compile time a Perl GDAL dll (for the bindings) is created and put into some location, which is known to Perl and Perl knows how to find it in runtime. Assuming I've built the binaries, then in runtime (when the GDAL module is called for) I think somebody (because the Perl GDAL module dll has a link to GDAL dll that was put there in compile time) asks for the GDAL dll's and they simply need to be available. Thus, if there is GDAL in the system (for example in c:\Program Files\Share\GDAL\bin) then that path needs to be in the PATH - I don't think there is a way around that. The PATH may be a temporary PATH set by somebody for Perl programs, but I think the policy should be that when you run GDAL.msi, the system default PATH is modified to have the path to the GDAL binaries. Best regards, Ari ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
\site-packages\osgeo\bin. Modify gdal.py to set os.environ['PATH'] = os.environ['PATH'] + ';' gdalInstallDir to modify the PATH to include that directory prior to importing _gdal.pyd. The PATH will be modified for the running process only, for the duration of that process. 3.Same as #2 but rather than modifying gdal.py to set the PATH variable, instead create a new Python extension module called _gdal_dll_helper.pyd. The job of this C extension module is simply to get gdal.dll and other DLLs loaded without resorting to modifying the system PATH which can sometimes have weird consequences (I can explain more if needed). The extension module would call the Windows SetDllDirectory http://msdn.microsoft.com/en-us/library/ms686203%28v=vs.85%29.aspx function, call LoadLibrary to explicitly load gdal17.dll into the current process, then call SetDllDirectory again to set the DLL directory back to what it was previously. Then, when gdal.py wants to load _gdal.pyd, gdal17.dll is already loaded and the binding succeeds. 4.Statically link all necessary code to _gdal.pyd, _gdal_array.pyd, etc. This would eliminate the need to store the GDAL DLLs on the machine at all, as all code would be in the bindings .pyd DLLs. This would likely be a big job, and probably not even possible given the variety of libraries that GDAL leverages. I know #2 and #3 sound scary but they can be done cleanly. I currently use a variation of #3 in my own project that embeds GDAL and its Python bindings. You also asked: The individual packages may specify the required minimum version of the referred packages loaded by the .NET runtime. Is this something that can be done with the python environment as well? As far as I know, there is not really a clean, standardized way to do that. There are some innovations related to the easy_install thing I mentioned earlier, but nothing has become standard in Python itself, at least with the 2.x releases. Jason From: Tamas Szekeres [mailto:szeker...@gmail.com] Sent: Thursday, January 06, 2011 7:24 AM To: Jason Roberts Cc: gdal-dev@lists.osgeo.org; Christopher Barker Subject: Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0 Jason, I appreciate the expertise for all of you along with this thread, I could already gather quite some useful information from here for this reason. I must mention that my programming practice in Python can be considered as zero, this is the main reason that my issues may have trivial solutions for the hardcore pythonists but not trivial to me. Apologies for this inconvenience :-) Getting back to the original topic, you mention that the gdal binaries should be installed somewhere an set PATH, GDAL_DATA, PROJ_LIB and GDAL_DRIVER_PATH as a systemwide setting. This is where the problems of mine are starting. Modifying the PATH globally is a bad practice in 99% of the cases. The only case I'm aware of which may not be a problem when we make sure that only one version of such files (dll-s and executables) will ever be installed to a particular system. But this is not the case with the gdal binaries as I would expect at least a development or a stable version (and their x86/x64 variants) to coexist which should be used by the same user. The same problem may arise when we would like to install multiple versions to the site packages directory, how the versions of the files are maintained by the python runtime? In this regard I could mention something like what have been done with the .NET framework with the multiple versions of the packages installed simultaneously in the global assembly cache. The individual packages may specify the required minimum version of the referred packages loaded by the .NET runtime. Is this something that can be done with the python environment as well? (As opposed to this, the dumb solution of having a starting script to open a command prompt (and setting PYTHONPATH properly) would ensure multiple versions to be used at the same time, since those settings are applyed to the cmd process solely.) Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
2011/1/6 Jason Roberts jason.robe...@duke.edu 2.Build an installation package as above. Have it install the GDAL DLLs as a subdirectory of the osgeo directory, e.g. C:\PythonXY\Lib\site-packages\osgeo\bin. Modify gdal.py to set os.environ['PATH'] = os.environ['PATH'] + ';' gdalInstallDir to modify the PATH to include that directory prior to importing _gdal.pyd. The PATH will be modified for the running process only, for the duration of that process. 3.Same as #2 but rather than modifying gdal.py to set the PATH variable, instead create a new Python extension module called _gdal_dll_helper.pyd. The job of this C extension module is simply to get gdal.dll and other DLLs loaded without resorting to modifying the system PATH which can sometimes have weird consequences (I can explain more if needed). The extension module would call the Windows SetDllDirectoryhttp://msdn.microsoft.com/en-us/library/ms686203%28v=vs.85%29.aspxfunction, call LoadLibrary to explicitly load gdal17.dll into the current process, then call SetDllDirectory again to set the DLL directory back to what it was previously. Then, when gdal.py wants to load _gdal.pyd, gdal17.dll is already loaded and the binding succeeds. I know #2 and #3 sound scary but they can be done cleanly. I currently use a variation of #3 in my own project that embeds GDAL and its Python bindings. Jason, Good writeup, thanks for that. You mention #2 and #3 which is exactly the same I have in my mind. Already posted a code for #2 related to the csharp bindings to demonstrate this option. #2 would be fairly transparent for the user (no additional modules imported), the only thing we could add is how gdal.py would find out the install location of the gdal files. This may probably set in the registry added by the installer. Not sure how this addition would affect the gdal binding on other platforms (if we implement this in gdal) #3 seems to be better for gdal (doesn't require to modify the bindings) however the user will require to use an additional module (which is not required normally when using gdal.py) I would personally vote on having an extension (like #3) which is imported by gdal.py in case if it is installed. If this extension is not installed, gdal.py would work as before. This extension would scan the registry to find out the install location of the corresponding files (probably based on an unique key) and perform the required actions to make the dll-s loadable. Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Tamas, I think with either option #2 or #3, gdal.py will have to be modified in order for everything to be invisible to the user. In #2, gdal.py would have to modify the PATH variable. In #3, it would have to import the _gdal_dll_helper module. In either case, it would just be a few lines of code at the top of the file that occurs before any other GDAL extension modules are loaded (_gdal, _gdal_array, etc). It would also need to be done for other modules such as ogr.py, osr.py, or whatever also imports GDAL extension modules (unless those ones also import gdal.py first). So #3 is not better by virtue of not having to modify the bindings; it does have to modify the bindings. But #3 is appealing because setting the PATH from Python code sometimes has weird issues. For example, when I tried it in my code, it produced a weird problem when I tried to use the nose Python testing framework to run test cases that involved calling Python modules from GDAL and ArcGIS. The problem there may ultimately be a Microsoft issue in which the environment settings for a process are maintained by the Microsoft C runtime library (msvcrt*.dll) and Python, ArcGIS, and GDAL may use different versions of that library. I do not want to digress into this here, but suffice to say, I prefer #3 to #2 because it did not have this problem. Here is an outline of tasks. We probably should put it in an RFC to keep track. 1.Modify the makefiles for your SDK so that it runs release--dev\gdal\swig\python\setup.py with the bdist --formats=wininst option. This will produce an installation program such as gdal-1.7.3.win32-py2.5.exe. This is what the user will run to install the Python bindings together with a private copy of the GDAL DLLs used just by those bindings. On Python 2.6 and later, we probably want bdist --formats=msi to produce a .msi file rather than a .exe, because Windows likes .msi files much better than .exes for security purposes. IIRC, Python 2.5 and earlier do not have the msi option. 2.Modify release--dev\gdal\swig\python\setup.py to include the GDAL DLLs and data files in the data_files list that is passed to the setup() function. Make sure it is only done for Windows (Python code can check that). The goal is to have the installation program create the following kind of installation: Existing Python bindings: C:\PythonXX\Lib\site-packages\osgeo\gdal.py C:\PythonXX\Lib\site-packages\osgeo\ogr.py C:\PythonXX\Lib\site-packages\osgeo\... C:\PythonXX\Lib\site-packages\osgeo\_gdal.pyd C:\PythonXX\Lib\site-packages\osgeo\_ogr.pyd C:\PythonXX\Lib\site-packages\osgeo\... New bin, data, and plugins directories, which are private to this installation of the bindings: C:\PythonXX\Lib\site-packages\osgeo\bin\gdal17.dll C:\PythonXX\Lib\site-packages\osgeo\bin\proj.dll C:\PythonXX\Lib\site-packages\osgeo\bin\... C:\PythonXX\Lib\site-packages\osgeo\bin\gdal-data\coordinate_axis.csv C:\PythonXX\Lib\site-packages\osgeo\bin\gdal-data\... C:\PythonXX\Lib\site-packages\osgeo\bin\gdal-plugins\ogr_PG.dll C:\PythonXX\Lib\site-packages\osgeo\bin\gdal-plugins\... 3.Introduce a new file release--dev\gdal\swig\python\extensions\_gdal_dll_loader.cpp. This file will call GetDllDirectory to find the current DLL directory setting, call SetDllDirectory to C:\PythonXX\Lib\site-packages\osgeo\bin (or wherever the Python instance is installed), call LoadLibrary(gdal17.dll), and call SetDllDirectory back to what it was before. There are more details to consider. For example, we might want to have it call LoadLibrary first to see if it can load the GDAL DLL from the PATH, allowing the user to use their own GDAL DLLs without having to overwrite the ones in the Python directory. Or we might not want to do that, with the idea that the GDAL Python bindings must be tightly coupled to a particular version of GDAL's DLLs. 4.Modify setup.py to compile _gdal_dll_loader.cpp, only for Windows. This will result in _gdal_dll_loader.pyd alongside the other .pyd files. 5.Modify gdal.py, osr.py, and so on to do something like this: import sys if sys.platform == 'win32': # TODO: add check for Windows x64 as well try: import osgeo._gdal_dll_loader # As part of this, Python will call an init function inside. That function will do the SetDllDirectory, etc. except: pass 6.Do whatever is necessary to ensure the GDAL_DATA etc are properly set up inside GDAL. I can't remember if this will just work using directories named gdal-data, gdal-plugins, or if it is necessary for the _gdal_dll_loader.pyd to call some GDAL functions to make it happen. Does that sound like the right direction? Jason From: Tamas Szekeres [mailto:szeker...@gmail.com] Sent: Thursday, January 06, 2011 11:07 AM To: Jason Roberts Cc: gdal-dev@lists.osgeo.org Subject: Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0 2011/1/6 Jason Roberts jason.robe
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Lots of stuff in this thread, I'll try to contribute what I can inline: On 1/6/11 2:15 AM, Ari Jolma wrote: When I started there was only ActivePerl for Windows that was usable, but it turned out impossible for me to use because they use MS compilers. Now there is Strawberry Perl, which I might try again (or pay someone to try). It is good because it uses MinGW. The situation is a bit different for Python -- the Python.org binaries are almost universal (and I think the ActiveState ones are compatible). So it's the obvious choice to build against. They are built with MS compilers, but you can compile extensions with MinGW (at least for 2.6 and above). However, ideally the MS compilers are users -- there are freely available versions that work (not Open source though!) Then there could also be a more end-user type Windows installation package project for GDAL, which installs the stuff into some standard location (c:\Program Files\share?). I think so too, but it's a bit unclear to me what's best as far as the various language bindings go...given all the versions of Python (and I assume others) it's a bit messy. In fact Geoinformatica should be simply installable through Perl's cpan application - but that does not even work in Linux yet :( (the reason is that the GDAL Perl module installation script is not intelligent enough and the version of the module in CPAN is horribly old and I really suck at managing that) Wow! given how much folks rave about CPAN, it sounds like PyPi is better in this regard, at least for binary packages. On 1/6/11 4:00 AM, Jan Hartmann wrote: Three months ago I put an announcement on the MapServer-Users list about a shell script I wrote to compile the whole MapServer-GDAL-PostGIS-PLR suite, with and without PHP/Python, Did you provide Python, to use an existing python.org build? Since then I got no (zero) replies, so I don't know whether it doesn't work, isn't interesting, or people just didn't look at it. well, for what it's worth, on LInux, users can go a long way with distro packages and the plain old ./configure; make.. dance. Binaries on Windows are in much higher demand. On 1/6/11 4:24 AM, Tamas Szekeres wrote: Modifying the PATH globally is a bad practice in 99% of the cases. really? for command line tools, I do it all the time! It seems the ay to go if you want it to just work. Maybe it makes for more dll hell? I would expect at least a development or a stable version (and their x86/x64 variants) to coexist which should be used by the same user. This is mostly an issue for developers, and you are smart enough to take care of yourselves... As for me, I usually have a system wide default for things like GDAL on the PATH, and then will have a startup script or something that re-sets things if I want to run test versions, or I just call the exe I want explicitly Anyway, putting a binary in a standard location is separate from having an installer modify the users PATH -- that second step should be optional, or just leave it for the user to do manually if they want. The same problem may arise when we would like to install multiple versions to the site packages directory, how the versions of the files are maintained by the python runtime? Python is very weak in that regard, There are three ways to handl eit now: 1)a roll-your-open version selection mechanism -- wxPython and I think PyGTK do this. 2) virtualenv, or similar -- they set up a virtual python install, so that you can have multiple installs, each with different versions of packages 3) setuptools provides some mechanism for egg version selection, though i've never really figure out how to use it. I think (2) is the best recommendation -- the nice thing about it is that the packager doesn't have to do anything special -- it's handled by how the user installs a package -- if they use setup.py or easy_install, or pip, it just works -- maybe not with an msi, though. So this points to building binary eggs, ideally. The individual packages may specify the required minimum version of the referred packages loaded by the .NET runtime. Is this something that can be done with the python environment as well? That's kind of what setuptools provides, but it's pretty kludgy. (As opposed to this, the dumb solution of having a starting script to open a command prompt (and setting PYTHONPATH properly) would ensure multiple versions to be used at the same time, since those settings are applied to the cmd process solely.) I suspect that's how virtualenv works, more or less. On 1/6/11 5:01 AM, Tamas Szekeres wrote: I did mention to set the environment for the cmd process solely, and not by a systemwide setting. In this regard it doesn't matter how many python runtime is installed on a particular system, the only issue is to refer to the desired one (by setting the PATH to the python executable or using absolute path when running python) from the command
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Tamas, In Perl the programmer can postpone loading the external DLL required by a module. Thus a similar approach is viable in Perl. But that's not how it's usually done. If you have a DLL in a system that is meant for shared use then it should be in the system PATH. Thus I believe the policy should be to set the PATH if GDAL is installed into a system (as such - not as a part of something else). Only if a package installs a private GDAL, it should be kept out of PATH and the package executables need to take care of setting the PATH or other env vars as appropriate (in Geoinformatica I do this - the Geoinformatica installer is basically a file copy and no env vars are changed). Ari 6.1.2011 16:50, Tamas Szekeres kirjoitti: Ari The only wokaround which is satisfactory to me is to set the required enviroment at run time before the gdal bindings are used. In csharp we can do something like: string path = Environment.GetEnvironmentVariable(PATH); if (!path.Contains(gdal_bin_path)) { Environment.SetEnvironmentVariable(PATH, gdal_bin_path + ; + path); } Gdal.SetConfigOption(GDAL_DRIVER_PATH, gdal_driver_path); But it requires that loading the gdal dll is happening when the interface is used actually (in Gdal.SetConfigOption with the code above). This is the case with the .NET runtime, but I'm not sure how do we stand with the other bindings in this regard. Futher problem with this solution is that this startup code should be placed in the user's script and not sure how this could be implemented in the gdal bindings itself. Best regards, Tamas 2011/1/6 Ari Jolma ari.jo...@gmail.com mailto:ari.jo...@gmail.com On 01/06/2011 03:25 PM, Tamas Szekeres wrote: Assuming we install the gdal-perl modules in a standard location (not sure where it is), do we have a common mechanism in the perl runtime to find the dependent dlls without having to violate system wide settings (like the PATH environment variable)? :) I'm not completely satisfied (I'm also not sure I really understand the whole f* issue) with how Perl searches for dll's - especially dll import libs - in Windows. But that's configure/compile time issue. In compile time a Perl GDAL dll (for the bindings) is created and put into some location, which is known to Perl and Perl knows how to find it in runtime. Assuming I've built the binaries, then in runtime (when the GDAL module is called for) I think somebody (because the Perl GDAL module dll has a link to GDAL dll that was put there in compile time) asks for the GDAL dll's and they simply need to be available. Thus, if there is GDAL in the system (for example in c:\Program Files\Share\GDAL\bin) then that path needs to be in the PATH - I don't think there is a way around that. The PATH may be a temporary PATH set by somebody for Perl programs, but I think the policy should be that when you run GDAL.msi, the system default PATH is modified to have the path to the GDAL binaries. Best regards, Ari ___ gdal-dev mailing list gdal-dev@lists.osgeo.org mailto:gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
On 1/6/11 12:13 PM, Ari Jolma wrote: If you have a DLL in a system that is meant for shared use then it should be in the system PATH. Thus I believe the policy should be to set the PATH if GDAL is installed into a system I notice on my system, the dll is gdal17.dll That is, the version is part of the file name, so there shouldn't be a problem with different versions installed in the PATH. WE could even use a longer filename, like *nix systems, we're not restricted to 8.3 anymore, are we? gdal-1.7.1-a.dll for instance. Though maybe setting up the build to do that is more trouble that it's worth. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Chris, Here are some comments on specific parts of your mail: me neither, with Python (or anything Windows, for that matter). Maybe Jason knows better, but I *think* we should be OK with a standard location for GDAL. But could you have: Program Files\GDAL\1.6\gdal.dll and Program Files\GDAL\1.6\gdal.dll and have one Python binding find 1.6, and one find 1.7 -- I'm not sure how to do that, but it SHOULD be doable! OH what i'd give for a symlink! Those would be reasonable locations for GDAL to live if the GDAL team decided to distribute the GDAL binaries using an installer that adhered to the best practices on Windows. But putting them there would not mean they would be automatically added to the PATH. The GDAL Python bindings would still have to use one of the options I mentioned to load the binaries there (e.g. modify the PATH). Incidentally, Windows does support both symbolic and hard links, although it is not widely known, and I'm not sure I'd recommend their use for this particular problem. I think 2.5 is MSVC2003 (but that may be 2.4, my memory is fading...). I believe both 2.4 and 2.5 used MSVC2003. Great summary -- thanks! And you clearly have a better grasp that I on Windows dll hell. I used to work for Microsoft, back in the day... :-) Modify gdal.py to set os.environ['PATH'] = os.environ['PATH'] + ';' gdalInstallDir to modify the PATH to include that directory prior to importing _gdal.pyd. The PATH will be modified for the running process only, for the duration of that process. I didn't think that was required, if the dll is in the same place as the *.pyd, but maybe it is. And it seems it would work fine. Might it mess things up if someone where to do a system call from Python? Also, would that support py2exe? Unfortunately, it is not sufficient to simply put the GDAL DLLs in the same directory as the GDAL .pyd files. When the .pyd tries to load the GDAL DLL through implicit linking, Windows checks the locations described in http://msdn.microsoft.com/en-us/library/7d83bc18%28v=vs.80%29.aspx. You are probably thinking of the first rule in that list, the directory where the executable module for the current process is located, i.e. the .exe file. In most cases, this is C:\PythonXY\python.exe. C:\PythonXY is probably not an appropriate location for the GDAL DLLs. Plus it would not work for other executables that host the Python interpreter, such as C:\Program Files\ArcGIS\Bin\ArcMap.exe. Incidentally, the Python interpreter itself is in a DLL that Python installs in C:\Windows\system32, which is always very high up in the PATH. GDAL could put its binaries there. But that might be very risky because GDAL ships a lot of DLLs that other apps might also use. For example, GDAL optionally includes xerces-c, which MATLAB and ArcGIS also use, among others. Plunking GDAL's own copy of xerces-c into the PATH will screw up those apps. I wonder what is donefor theones here: http://www.lfd.uci.edu/~gohlke/pythonlibs/ I have been meaning to look at that since you originally sent it. It probably gives an example of the kinds of stuff I mentioned. I wonder if it is statically linked. Rgdal is one example where someone has statically linked GDAL into their own project. could all that be done with some ctypes calls, rather than another extension? Yes I believe you are absolutely right. I always forget about ctypes because I'm working on something that must be compatible back to Python 2.4 and ctypes was not included in 2.4. Jason ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
On 1/6/11 12:31 PM, Jason Roberts wrote: Here are some comments on specific parts of your mail: Program Files\GDAL\1.6\gdal.dll and Program Files\GDAL\1.6\gdal.dll Those would be reasonable locations for GDAL to live if the GDAL team decided to distribute the GDAL binaries using an installer that adhered to the best practices on Windows. I *think* we're heading for a consensus to do that, but not many people have been on this thread. Incidentally, Windows does support both symbolic and hard links, although it is not widely known, and I'm not sure I'd recommend their use for this particular problem. I've heard that, but never seen it done. You are probably thinking of the first rule in that list, the directory where the executable module for the current process is located, i.e. the .exe file. In most cases, this is C:\PythonXY\python.exe. C:\PythonXY is probably not an appropriate location for the GDAL DLLs. Plus it would not work for other executables that host the Python interpreter, such as C:\Program Files\ArcGIS\Bin\ArcMap.exe. What about C:\PYTHON26\libs ? or is that for the main python distro only? Incidentally, the Python interpreter itself is in a DLL that Python installs in C:\Windows\system32, which is always very high up in the PATH. GDAL could put its binaries there. But that might be very risky because GDAL ships a lot of DLLs that other apps might also use. I agree -- and Tamas would never go for it! ;-) I wonder what is donefor theones here: http://www.lfd.uci.edu/~gohlke/pythonlibs/ I have been meaning to look at that since you originally sent it. let us now what you find -- I really can't spend any more time on this right now! Rgdal is one example where someone has statically linked GDAL into their own project. so we know it can be done -- but with all the third party libs (xerces, etc, etc) that could be ugly/impossible. could all that be done with some ctypes calls, rather than another extension? Yes I believe you are absolutely right. I always forget about ctypes because I'm working on something that must be compatible back to Python 2.4 and ctypes was not included in 2.4. I've never actually used it, but it does seem made for this sort of thing. As for versions, I don't know if GDAL has a policy about what versions to support, but I'd think we could dump 2.4 (and maybe even 2.5...), and that means only one MS compiler to support. Thanks for your work on this, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
2011/1/6 Christopher Barker chris.bar...@noaa.gov I notice on my system, the dll is gdal17.dll That is, the version is part of the file name, so there shouldn't be a problem with different versions installed in the PATH. WE could even use a longer filename, like *nix systems, we're not restricted to 8.3 anymore, are we? gdal-1.7.1-a.dll Chris, This is where a typical dll hell problem is starting. The application is happy to load a common dll let's say a gdal17.dll, zlib1.dll, libcurl.dll whatever, but is not the same at it is expected to be (for example it depends on different CRT libraries). The issue is that you rarely get a clear error message what the problem is, but your application is failing with access violations in certain conditions. To make sure about the issue I did a quick test on my devserver with a whereis.bat containing the following script, trying to find the location of some common dlls (used by gdal): @for %%e in (%PATHEXT%;.dll) do @for %%i in (%1%%e) do @if NOT %%~$PATH:i== echo %%~$PATH:i E:\buildswhereis zlib1 C:\Program Files (x86)\Mono-1.2.6\bin\zlib1.dll E:\buildswhereis ssleay32 C:\Program Files (x86)\Subversion\ssleay32.dll Depending on the location of my entry in the PATH may break both applications or these files may break mine. Not sure this causes an issue in this particular case but, it makes the things fragile well enough. Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
On 11-01-06 04:42 PM, Christopher Barker wrote: On 1/6/11 12:31 PM, Jason Roberts wrote: Here are some comments on specific parts of your mail: Program Files\GDAL\1.6\gdal.dll and Program Files\GDAL\1.6\gdal.dll Those would be reasonable locations for GDAL to live if the GDAL team decided to distribute the GDAL binaries using an installer that adhered to the best practices on Windows. I *think* we're heading for a consensus to do that, but not many people have been on this thread. Jason / Christopher, I have no objection to using \Program Files\GDAL\major.minor\ as a standard location for GDAL stuff on windows. If this is done, the GDAL data search location should be compiled in for this location on windows, much as it defaults to /usr/share/gdal (or /usr/local/share/gdal) on linux. Note that this means point releases cannot coexist on a system. I think that is mostly a good thing, in that upgrading to a new point release should not change the ABI and should be a transparent change for any applications. That is also why we do not encode the point release values in the DLL name. That is the GDAL17.DLL from GDAL 1.7.3 should be a drop in replacement for the GDAL17.DLL from GDAL 1.7.2. If the DLL name was more specific then applications would be forced to relink to use the new version. On the other hand, we do want major relases (1.7.x vs. 1.8.x) to coexist and it is decision that applications need to make which they want to use. So the proposed structure handles that well. I'm going to stay out of the Python specific aspects. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Programmer for Rent ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
On 1/6/11 1:51 PM, Tamas Szekeres wrote: 2011/1/6 Christopher Barker chris.bar...@noaa.gov This is where a typical dll hell problem is starting. The application is happy to load a common dll let's say a gdal17.dll, zlib1.dll, libcurl.dll whatever, but is not the same at it is expected to be (for example it depends on different CRT libraries). yup -- it seems long, descriptive file names would help, but that doesn't seem to be the Windows way... OS_X hard-codes the path to shared libs when linking, which causes its own problems, but not these ones. I think Jason has some good ideas, though. To make sure about the issue I did a quick test on my devserver with a whereis.bat containing the following script, trying to find the location of some common dlls (used by gdal): @for %%e in (%PATHEXT%;.dll) do @for %%i in (%1%%e) do @if NOT %%~$PATH:i== echo %%~$PATH:i E:\buildswhereis zlib1 C:\Program Files (x86)\Mono-1.2.6\bin\zlib1.dll E:\buildswhereis ssleay32 C:\Program Files (x86)\Subversion\ssleay32.dll so your gdal depends on Mono's zlib and Subversion's ssleay? that does seem fragile! Or do you mean that you have these somewhere for gdal, and they are ALSO in those other locations? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Frank, Thanks for sharing your opinion. I do have one question that I hope you will weigh in on. Which of the following two options seems better to you: 1. The GDAL libraries (possibly accompanied with executable programs) are installed as a separately from the GDAL Python bindings. The libraries are installed to \Program Files as you describe and the Python bindings are installed in the standard Python location. A Windows Python programmer wanting to use GDAL would perform two installs: one for the GDAL libraries, the other for the bindings. 2. The GDAL Python bindings include a private copy of the GDAL libraries (with no executable programs). These are installed to a private subdirectory with the bindings. A Windows Python programmer wanting to use GDAL only needs to perform one install: the bindings. #1 is essentially how things are done now, just not following Windows best practices (not using \Program files) and without any automation to ease the process (no regularly maintained installer for the Python bindings). I was proposing #2, basically under the argument that a Windows Python programmer who needs to use GDAL will rarely ever need to use GDAL for some other purpose, and therefore not want to install and think about a separate installation of the GDAL libraries themselves. But if you don't believe that, or think it is important that the libraries remain distinct from the bindings in this scenario for some other reason, then we would appreciate your opinion. Thanks, Jason -Original Message- From: gdal-dev-boun...@lists.osgeo.org [mailto:gdal-dev-boun...@lists.osgeo.org] On Behalf Of Frank Warmerdam Sent: Thursday, January 06, 2011 5:01 PM To: gdal-dev@lists.osgeo.org Subject: Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0 On 11-01-06 04:42 PM, Christopher Barker wrote: On 1/6/11 12:31 PM, Jason Roberts wrote: Here are some comments on specific parts of your mail: Program Files\GDAL\1.6\gdal.dll and Program Files\GDAL\1.6\gdal.dll Those would be reasonable locations for GDAL to live if the GDAL team decided to distribute the GDAL binaries using an installer that adhered to the best practices on Windows. I *think* we're heading for a consensus to do that, but not many people have been on this thread. Jason / Christopher, I have no objection to using \Program Files\GDAL\major.minor\ as a standard location for GDAL stuff on windows. If this is done, the GDAL data search location should be compiled in for this location on windows, much as it defaults to /usr/share/gdal (or /usr/local/share/gdal) on linux. Note that this means point releases cannot coexist on a system. I think that is mostly a good thing, in that upgrading to a new point release should not change the ABI and should be a transparent change for any applications. That is also why we do not encode the point release values in the DLL name. That is the GDAL17.DLL from GDAL 1.7.3 should be a drop in replacement for the GDAL17.DLL from GDAL 1.7.2. If the DLL name was more specific then applications would be forced to relink to use the new version. On the other hand, we do want major relases (1.7.x vs. 1.8.x) to coexist and it is decision that applications need to make which they want to use. So the proposed structure handles that well. I'm going to stay out of the Python specific aspects. Best regards, -- ---+ -- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Programmer for Rent ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
2011/1/6 Christopher Barker chris.bar...@noaa.gov yup -- it seems long, descriptive file names would help, but that doesn't seem to be the Windows way... It would be true, but it's not a common practice of the libraries currently (assuming we keep the original dll name of each dependency). However we could probably rename our custom dlls with a random name during the build process. Pretty odd, but workable. so your gdal depends on Mono's zlib and Subversion's ssleay? that does seem fragile! Or do you mean that you have these somewhere for gdal, and they are ALSO in those other locations? I must say a common gdal build (either FWTools, OSGeo4w, ms4w whatever) may use dll-s with the same names as many other libraries a user may install. In my particular case mono and subversion is just a live example. but we could find many others that use zlib libpng libcurl whatever (these are widely used libraries in the world). I don't actually suffer from this problem because I never rely on the PATH environment to load the GDAL dependencies and I would evangelize against this practice when trying to find out a reasonable setup approach here also. Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
On 11-01-06 05:10 PM, Jason Roberts wrote: Frank, Thanks for sharing your opinion. I do have one question that I hope you will weigh in on. Which of the following two options seems better to you: 1. The GDAL libraries (possibly accompanied with executable programs) are installed as a separately from the GDAL Python bindings. The libraries are installed to \Program Files as you describe and the Python bindings are installed in the standard Python location. A Windows Python programmer wanting to use GDAL would perform two installs: one for the GDAL libraries, the other for the bindings. 2. The GDAL Python bindings include a private copy of the GDAL libraries (with no executable programs). These are installed to a private subdirectory with the bindings. A Windows Python programmer wanting to use GDAL only needs to perform one install: the bindings. #1 is essentially how things are done now, just not following Windows best practices (not using \Program files) and without any automation to ease the process (no regularly maintained installer for the Python bindings). I was proposing #2, basically under the argument that a Windows Python programmer who needs to use GDAL will rarely ever need to use GDAL for some other purpose, and therefore not want to install and think about a separate installation of the GDAL libraries themselves. But if you don't believe that, or think it is important that the libraries remain distinct from the bindings in this scenario for some other reason, then we would appreciate your opinion. Jason, I don't have a strong position on this, but I would note that beyond real Python programmers wanting to use the bindings, they are also needed just to run the python commandline utilities (such as rgb2pct.py). So there is definately a large body of folks who would benefit from having working python for the python utilities, and the regular commandline executables. Another benefit of even Python using GDAL from a standard location is things like format plugins could be easily handled in one place. Actually, the more I think about it, the more I prefer even the Python bindings to use the GDAL under \Program Files\GDAL. If the Pythonistas really want to have their own copy of GDAL then we really don't need to have this conversation. They can do their own thing, in their own place and there is no need for meaningful cooperation with the core project. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Programmer for Rent ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
2011/1/6 Jason Roberts jason.robe...@duke.edu So #3 is not better by virtue of not having to modify the bindings; it does have to modify the bindings. But #3 is appealing because setting the PATH from Python code sometimes has weird issues. For example, when I tried it in my code, it produced a weird problem when I tried to use the “nose” Python testing framework to run test cases that involved calling Python modules from GDAL and ArcGIS. The problem there may ultimately be a Microsoft issue in which the environment settings for a process are maintained by the Microsoft C runtime library (msvcrt*.dll) and Python, ArcGIS, and GDAL may use different versions of that library. I do not want to digress into this here, but suffice to say, I prefer #3 to #2 because it did not have this problem. This is quite a common issue the getenv/_putenv CRT functions operate only on the data structures accessible to the run-time library (CRT) and not on the environment segment created for the process by the operating system. In this regard the libraries work only on a snapshot of the variables that have been set during the CRT startup. In this case we should probably find a function that invokes the SetEnvironmentVariablehttp://msdn.microsoft.com/en-us/library/ms686206%28v=vs.85%29.aspxAPI instead. 1.Modify the makefiles for your SDK so that it runs release--dev\gdal\swig\python\setup.py with the “bdist --formats=wininst” option. This will produce an installation program such as gdal-1.7.3.win32-py2.5.exe. This is what the user will run to install the Python bindings together with a private copy of the GDAL DLLs used just by those bindings. On Python 2.6 and later, we probably want “bdist --formats=msi” to produce a .msi file rather than a .exe, because Windows likes .msi files much better than .exes for security purposes. IIRC, Python 2.5 and earlier do not have the msi option. This can be done fairly easily. Does this mean we would require each build for multiple python versions? 2.Modify release--dev\gdal\swig\python\setup.py to include the GDAL DLLs and data files in the data_files list that is passed to the setup() function. Make sure it is only done for Windows (Python code can check that). The goal is to have the installation program create the following kind of installation: Hmmm. I'm keen to avoid any local modifications in the compiled files throughout the build system. Do we have some option to load this infromation from an external file (which is probably loaded by setup.py if exists)? Or I must write a custom setup2.py containing this customization. 3.Introduce a new file release--dev\gdal\swig\python\extensions\_gdal_dll_loader.cpp. This file will call GetDllDirectory to find the current DLL directory setting, call SetDllDirectory to C:\PythonXX\Lib\site-packages\osgeo\bin (or wherever the Python instance is installed), call LoadLibrary(gdal17.dll), and call SetDllDirectory back to what it was before. There are more details to consider. For example, we might want to have it call LoadLibrary first to see if it can load the GDAL DLL from the PATH, allowing the user to use their own GDAL DLLs without having to overwrite the ones in the Python directory. Or we might *not* want to do that, with the idea that the GDAL Python bindings must be tightly coupled to a particular version of GDAL’s DLLs. That seems to be quite complicated, not sure it that's working for those libraries using LoadLibrary binding at run time. Wouldn't that be better getting back to add the location to the PATH of the process in a loader.py (using SetEnvironmentVariable). 5.Modify gdal.py, osr.py, and so on to do something like this: import sys if sys.platform == 'win32': # TODO: add check for Windows x64 as well try: import osgeo._gdal_dll_loader # As part of this, Python will call an “init” function inside. That function will do the SetDllDirectory, etc. except: pass Such changes can be done in the SWIG interface files easily. 6.Do whatever is necessary to ensure the GDAL_DATA etc are properly set up inside GDAL. I can’t remember if this will “just work” using directories named gdal-data, gdal-plugins, or if it is necessary for the _gdal_dll_loader.pyd to call some GDAL functions to make it happen. I guess this could also be set form a python script (probably from loader.py) Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
On 06-01-2011 21:42, Christopher Barker wrote: On 1/6/11 12:31 PM, Jason Roberts wrote: Here are some comments on specific parts of your mail: Program Files\GDAL\1.6\gdal.dll and Program Files\GDAL\1.6\gdal.dll Those would be reasonable locations for GDAL to live if the GDAL team decided to distribute the GDAL binaries using an installer that adhered to the best practices on Windows. I *think* we're heading for a consensus to do that, but not many people have been on this thread. Hi, I have been following this with attention and sometimes I thought I had something to say but than it has many python specific issues, which I'm not versed at all. My experience with other installers is GMT Mirone. Here I simply put the gdal dll and ALL its dependencies (determined by careful Dependency Walker analysis) in the same directory as the other exes. Since this must be in the PATH, the gdal package goes along. But this solution normally doesn't care about the gdal exes. Also a word about the best practice on Windows. I really don't see anything not even good in that practice to put them in Program Files. Having directories with blanks in their name give nothing but future problems when running command line programs (I have seen that happen many times). The GMT Mirone installs propose as default a C:\programs diectory. Let me remember that Program Files is not a unique name. For example in Portugues Win versions it is called O Meus Programas (Horror). Remember also that MS used to have Documents and Settings but now on Win7 it's only Users. Incidentally, Windows does support both symbolic and hard links, although it is not widely known, and I'm not sure I'd recommend their use for this particular problem. I've heard that, but never seen it done. It works pretty well, the command is mklink and works much like on *nix BUT exists only on Vista and Win7. Time to time, it has been raised also the hypothesis to put the dlls in windowes\system32. PLEASE, PLEASE don't even think on that. It's the easiest way of making the worst dll-hell. I learned that after spending many hours trying to understand why GDAL was not working in the classroom computers. It turned out to be an very old zlib1.dll put there by ArcS* Joaquim ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Frank, Thanks for your thoughts. Based on them, I'd recommend the following two things be created. 1. GDAL windows installation program, or at minimum, a wiki page that says how to install the GDAL libraries and utilities (executables and Python scripts) to \Program Files\GDAL\... Perhaps a quick compromise would be for Tamas's build system to produce a .zip that would have everything in a suitable directory structure and for wiki page to instruct the user just unzip this to \Program Files\GDAL\ 2. GDAL Python bindings installation program. This could be easily created using the standard Python distutils stuff I've been mentioning, and would install the bindings to the normal Python place. The bindings would ideally be modified to check for and explicitly load libraries from \Program Files\GDAL\... This would eliminate the need to modify the PATH variable. A Windows Python programmer would just install those two things and they're done. No questions about what goes where. No need to set any environment variables. People like me who embed GDAL in their own application would now have a new alternative: rather than including GDAL libraries we could instruct the user to download an official GDAL build and install it, and then just use it from there. What do you think? Jason -Original Message- From: Frank Warmerdam [mailto:warmer...@pobox.com] Sent: Thursday, January 06, 2011 5:49 PM To: Jason Roberts Cc: gdal-dev@lists.osgeo.org Subject: Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0 On 11-01-06 05:10 PM, Jason Roberts wrote: Frank, Thanks for sharing your opinion. I do have one question that I hope you will weigh in on. Which of the following two options seems better to you: 1. The GDAL libraries (possibly accompanied with executable programs) are installed as a separately from the GDAL Python bindings. The libraries are installed to \Program Files as you describe and the Python bindings are installed in the standard Python location. A Windows Python programmer wanting to use GDAL would perform two installs: one for the GDAL libraries, the other for the bindings. 2. The GDAL Python bindings include a private copy of the GDAL libraries (with no executable programs). These are installed to a private subdirectory with the bindings. A Windows Python programmer wanting to use GDAL only needs to perform one install: the bindings. #1 is essentially how things are done now, just not following Windows best practices (not using \Program files) and without any automation to ease the process (no regularly maintained installer for the Python bindings). I was proposing #2, basically under the argument that a Windows Python programmer who needs to use GDAL will rarely ever need to use GDAL for some other purpose, and therefore not want to install and think about a separate installation of the GDAL libraries themselves. But if you don't believe that, or think it is important that the libraries remain distinct from the bindings in this scenario for some other reason, then we would appreciate your opinion. Jason, I don't have a strong position on this, but I would note that beyond real Python programmers wanting to use the bindings, they are also needed just to run the python commandline utilities (such as rgb2pct.py). So there is definately a large body of folks who would benefit from having working python for the python utilities, and the regular commandline executables. Another benefit of even Python using GDAL from a standard location is things like format plugins could be easily handled in one place. Actually, the more I think about it, the more I prefer even the Python bindings to use the GDAL under \Program Files\GDAL. If the Pythonistas really want to have their own copy of GDAL then we really don't need to have this conversation. They can do their own thing, in their own place and there is no need for meaningful cooperation with the core project. Best regards, -- ---+ -- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Programmer for Rent ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
1.Modify the makefiles for your SDK so that it runs release--dev\gdal\swig\python\setup.py with the bdist --formats=wininst option. This will produce an installation program such as gdal-1.7.3.win32-py2.5.exe. This is what the user will run to install the Python bindings together with a private copy of the GDAL DLLs used just by those bindings. On Python 2.6 and later, we probably want bdist --formats=msi to produce a .msi file rather than a .exe, because Windows likes .msi files much better than .exes for security purposes. IIRC, Python 2.5 and earlier do not have the msi option. This can be done fairly easily. Does this mean we would require each build for multiple python versions? Yes, it would mean that. One build for Python 2.5, one for 2.6, one for 2.7, etc. This is how it is typically done. 2.Modify release--dev\gdal\swig\python\setup.py to include the GDAL DLLs and data files in the data_files list that is passed to the setup() function. Make sure it is only done for Windows (Python code can check that). The goal is to have the installation program create the following kind of installation: Hmmm. I'm keen to avoid any local modifications in the compiled files throughout the build system. Do we have some option to load this infromation from an external file (which is probably loaded by setup.py if exists)? Or I must write a custom setup2.py containing this customization. I was not saying that setup.py be forked and a special copy exist for windows. I was saying that setup.py should internally check whether it was being executed on Windows (check Python's sys.platform variable), and then perform special logic. But this may be moot because it sounded like Frank was not keen on my suggestion of enclosing a private copy of the GDAL DLLs with the Python bindings. If we avoid my suggestion then we are left with the idea of putting them in \Program Files\GDAL. In that case, we would not need to modify setup.py for this purpose. 3.Introduce a new file release--dev\gdal\swig\python\extensions\_gdal_dll_loader.cpp. This file will call GetDllDirectory to find the current DLL directory setting, call SetDllDirectory to C:\PythonXX\Lib\site-packages\osgeo\bin (or wherever the Python instance is installed), call LoadLibrary(gdal17.dll), and call SetDllDirectory back to what it was before. There are more details to consider. For example, we might want to have it call LoadLibrary first to see if it can load the GDAL DLL from the PATH, allowing the user to use their own GDAL DLLs without having to overwrite the ones in the Python directory. Or we might not want to do that, with the idea that the GDAL Python bindings must be tightly coupled to a particular version of GDAL's DLLs. That seems to be quite complicated, not sure it that's working for those libraries using LoadLibrary binding at run time. Wouldn't that be better getting back to add the location to the PATH of the process in a loader.py (using SetEnvironmentVariable). I do not think it would be good to tamper with setting the PATH. I am not certain, but I think Python's environment handler may call _putenv rather than SetEnvironmentVariable, or something like that, which can cause some problems. It might be better just to use SetDllDirectory. Also, as Chris pointed out, the Python ctypes module can be used to call win32 APIs. This would eliminate the need for a new _gdal_dll_loader module. Instead, the try/except block that I described below would just call win32 directly via ctypes. It could do the necessary SetDllDirectory/LoadLibrary/etc itself. 5.Modify gdal.py, osr.py, and so on to do something like this: import sys if sys.platform == 'win32': # TODO: add check for Windows x64 as well try: import osgeo._gdal_dll_loader # As part of this, Python will call an init function inside. That function will do the SetDllDirectory, etc. except: pass Such changes can be done in the SWIG interface files easily. Sounds good to me. 6.Do whatever is necessary to ensure the GDAL_DATA etc are properly set up inside GDAL. I can't remember if this will just work using directories named gdal-data, gdal-plugins, or if it is necessary for the _gdal_dll_loader.pyd to call some GDAL functions to make it happen. I guess this could also be set form a python script (probably from loader.py) If we follow the \Program Files\GDAL suggestion, then I further suggest that GDAL itself be modified to use that location as the default for GDAL_DATA, GDAL_PLUGINS, etc. That way the user does not have to set ANY environment variables for GDAL to work. That would be a major plus because, frankly, Windows users rarely have to set environment variables. Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
On 1/6/11 3:03 PM, Joaquim Luis wrote: Also a word about the best practice on Windows. I really don't see anything not even good in that practice to put them in Program Files. Having directories with blanks in their name give nothing but future problems when running command line programs (I have seen that happen many times). True, I'm still quite amazed how MS has ignored the command line for years. Sure, we geeks are a small fraction of Windows users, but a small fraction of a huge userbase is still a LOT. Perhaps that's why the standard place for Python is NOT in Program Files That being said, Program Files does seem to work fine for me most of the time. Time to time, it has been raised also the hypothesis to put the dlls in windowes\system32. PLEASE, PLEASE don't even think on that. That, we do have a consensus on! By the way, isn't there some new stuff about side by side installs or something -- that's supposed to help dll hell? On 1/6/11 2:49 PM, Frank Warmerdam wrote: I don't have a strong position on this, but I would note that beyond real Python programmers wanting to use the bindings, they are also needed just to run the python commandline utilities (such as rgb2pct.py). So there is definately a large body of folks who would benefit from having working python for the python utilities, and the regular commandline executables. yup. Also, I use gdal command line utilities and python bindings. It's actually really nice to know that they are using the exact same versions, as I can test stuff with the utilities and compare results. Another benefit of even Python using GDAL from a standard location is things like format plugins could be easily handled in one place. Yes, and then Perl, and who knows what else can share the same install. Actually, the more I think about it, the more I prefer even the Python bindings to use the GDAL under \Program Files\GDAL. +1 from me, too. I winder where Howard is? But anyway, the gdal PyPi site (which I think Howard wrote/maintained) says: Windows You will need the following items to complete an install of the GDAL Python bindings on Windows: * GDAL Windows Binaries The basic install requires the gdalwin32exe###.zip (### is the version number). Other files you see in the directory are for various optional plugins and development headers/include files. After downloading the zip file, extract it to the directory of your choosing. ** the problem here is that that zip file hasn'tbeen maintained for a while. * GDAL Python Bindings available at the Python Cheeseshop. An executable installer is available for both Python 2.4 or 2.5 or as a Python egg. ** this is pretty much what Jason is suggeting - a binary egg dependent on a certain gdal install. As explained in the README_EXE.txt file, after unzipping the GDAL binaries you will need to modify your system path and variables. If you're not sure how to do this, read the Microsoft KnowledgeBase doc 1. Add the installation directory bin folder to your system PATH, remember to put a semicolon in front of it before you add to the existing path. C:\gdalwin32-1.7\bin 2. Create a new user or system variable with the data folder from your installation. Name : GDAL_DATA Path : C:\gdalwin32-1.7\data It would be nice to remove these steps. Again, I think Jason's got some good ideas for that. By the way -- it is fairly common for python packages to include command line utilities. There are tools in distutils and setuptools to support that -- they get installed in a Scripts directory in the Python install. Should the Python-based gdal utilities go there? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Perhaps that's why the standard place for Python is NOT in Program Files Yes, there is wisdom. That being said, Program Files does seem to work fine for me most of the time. And for the (1 - most) times that it didn't work you knew how to fix it (enclosing the full path with ) but many people don't know it. By the way, isn't there some new stuff about side by side installs or something -- that's supposed to help dll hell? I think you are referring to the manifests but here also MS has changed their mind (aleluia). VC2010 doesn't have them anymore which IMO is a compelling reason to use this compiler whenever possible. Joaquim ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
On 11-01-06 06:43 PM, Jason Roberts wrote: Frank, Thanks for your thoughts. Based on them, I'd recommend the following two things be created. 1. GDAL windows installation program, or at minimum, a wiki page that says how to install the GDAL libraries and utilities (executables and Python scripts) to \Program Files\GDAL\... Perhaps a quick compromise would be for Tamas's build system to produce a .zip that would have everything in a suitable directory structure and for wiki page to instruct the user just unzip this to \Program Files\GDAL\ ... What do you think? Jason, I'm supportive, but not necessarily willing to take it as action item for myself. I would suggest building it as an installer .exe, perhaps using NSIS as I did for FWTools, or perhaps the method mentioned by Jurgen produces a nice installer. One question not discussed is whether GDAL should be minimalist or maximalist. That is, do we want to include as many formats as possible despite the fact that it drags in lots of supporting libraries? If we just use whatever decision OSGeo4W uses then we will have a fairly maximalist solution which is ok I suppose but might make integration of the resulting GDAL in other complex applications messy. I would like to suggest production of such an installer be done in such a way that the generating scripts live in SVN somewhere, and with instructions for the process in the wiki so it isn't all tied to one person (as FWTools was). Man, I'm really tempted to leap into this myself but I have so many other outstanding items right now. I am willing to assist in an effort by a small (2-3 people?) team. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Programmer for Rent ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Hi all, for what is worth also for me the simplicity of FWTOOLS wins. Best regards, andrea - Andrea Borruso email: aborr...@tin.it website: http://blog.spaziogis.it my 2.0 life: http://aborruso.spaziogis.it feed: http://feeds2.feedburner.com/Tanto 38° 7' 48 N, 13° 21' 9 E -- View this message in context: http://osgeo-org.1803224.n2.nabble.com/gdal-dev-FWTools-and-GDAL-1-7-0-tp5887121p5891689.html Sent from the GDAL - Dev mailing list archive at Nabble.com. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
On 1/5/11 3:41 AM, iomeneandrei wrote: for what is worth also for me the simplicity of FWTOOLS wins. FWTools is nice, but I think with OSgeo4win, not really important. However, for simplicity, a one-click installer for just GDAL/OGR for Windows, complete with command line tools and ready for use with the python bindings (and others language bindings?) would be great. I've found it painful to find appropriate Windows executables every time I've need to upgrade to the latest. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
2011/1/5 Christopher Barker chris.bar...@noaa.gov However, for simplicity, a one-click installer for just GDAL/OGR for Windows, complete with command line tools and ready for use with the python bindings (and others language bindings?) would be great. I've found it painful to find appropriate Windows executables every time I've need to upgrade to the latest. Supporting multiple vesions (development/stable branches/releases, x32/x64, multiple MSVC CRT dependencies) is quite a difficult task in a single installer. With regards to the development version it would also be reasonable to provide a build quite frequently to be in sync with the latest changes in trunk. These are the main reasons I've originally set up http://vbkto.dyndns.org/sdk/ to provide most of the resonable combinations as daily built packages. I also wanted to include these files in an installer (or multiple installers) but at the moment I don't see the real benefit of this over extracting a single zip package, since these libraries don't require significant preparation (like regkey entries) during the deployment. Any further consideration in this topic would be helpful, as I may have missed something that would also be important by a windows user. Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
On 11-01-05 12:44 PM, Tamas Szekeres wrote: I also wanted to include these files in an installer (or multiple installers) but at the moment I don't see the real benefit of this over extracting a single zip package, since these libraries don't require significant preparation (like regkey entries) during the deployment. Any further consideration in this topic would be helpful, as I may have missed something that would also be important by a windows user. Tamas, I agree with you, but it seems that an [OK] button (even if it doesn't do anything) makes Windows users feel so much better. :) Daniel -- Daniel Morissette http://www.mapgears.com/ ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
On 1/5/11 9:44 AM, Tamas Szekeres wrote: Supporting multiple vesions (development/stable branches/releases, x32/x64, multiple MSVC CRT dependencies) is quite a difficult task in a single installer. yes, a Major pain. I don't know that we need a single installer, but there is a lot to support. These are the main reasons I've originally set up http://vbkto.dyndns.org/sdk/ to provide most of the resonable combinations as daily built packages. Actually, that is a GREAT resource. At the moment, I can't remember why it's been difficult for me to use, but a few thoughts: First -- I use GDAL from Python and with the command line tools, so that's my perspective. 1) It would be nice to have binaries for the latest release front and center at the main GDAL site -- having to poke around to find Tamas's site is not a big deal, but not always obvious. 2) A standard install location would be good. As I've messed with this each time, I never know where stuff should go -- maybe installers would help with that 3) If there is a standard install location, then easy_install gdal (or setup.py build, or...) could work for the python bindings. Another option is to have a binary installer for the python bindings that includes gdal and the gdal utilities -- that would be great for users like me, but I don't know how common my use case is. In that case, you'd want to support a few recent pythons versions, the python.org binaries: 2.6, 2.7, 3.1 (maybe 2.5 too). One of the tricks here is which numpy to support, etc. numpy has been pretty good with binary compatibility lately (except for one mistake recently that was corrected) However, I DON'T want gdal to give me Python -- I use Python for too many other things for that. 4) it might be nice if the install location for the utilities got put on the user's PATH -- I don't know how hard that is in an installer -- Windows really sucks in that regard. I also wanted to include these files in an installer (or multiple installers) but at the moment I don't see the real benefit of this over extracting a single zip package, since these libraries don't require significant preparation (like regkey entries) during the deployment. An installer would better enforce a standard install location. You could also have a custom DOS box with a menu entry that set up the environment for the command line tools (maybe only PATH?), provide an uninstaller, and of course, give the Windows users a nice warm and fuzzy feeling. With regard to Python, an installer could see what Python the user has and install the bindings for that version. Not that I have any idea how to build that! Inno Setup is a very nice free installer, by the way. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
2011/1/5 Christopher Barker chris.bar...@noaa.gov 1) It would be nice to have binaries for the latest release front and center at the main GDAL site -- having to poke around to find Tamas's site is not a big deal, but not always obvious. Chris, With regards to the comment above, while I'm not sure about the objectives but I don't think the GDAL site would intend to be a hosting provider of various binary packages, the most reasonable thing is to put the references pointing the user to the correct locations which has already been done, see the Downloads section at http://www.gdal.org/ 2) A standard install location would be good. As I've messed with this each time, I never know where stuff should go -- maybe installers would help with that This doesn't seem to be decisive requirement to me. We may also create a definite location in the hard drive to host such files (which can be remembered later). Or some other folks may prefer installing these files along with their applications or keep such files in separate - project specific - directories. We may also have a requirement to deploy and run these applications from portable locations (ie. from CD or a flash drive). Another issue of an installer may be due to a single product key along with the setup which would prevent from installing multiple versions side by side in the same environment. 3) If there is a standard install location, then easy_install gdal (or setup.py build, or...) could work for the python bindings. I admit I don't have enough knowledge about the 'magic' tricks related to python-ish way installing applications. I expect that most 'magic' are implemented by copying the files at certain locations, setting some environment variables or regkey entries. However I might also consider running a custom application with gdal not necessarily be the responsibility of a GDAL package. You might also want to install python from a separate installer (either ActivePython, python.org whatever) and run the application direcly from a command prompt where the environments are set to run the gdal applications properly. Most of these packages provide the required .bat file to setup the environment this way. Another option is to have a binary installer for the python bindings that includes gdal and the gdal utilities -- that would be great for users like me, but I don't know how common my use case is. In that case, you'd want to support a few recent pythons versions, the python.org binaries: 2.6, 2.7, 3.1 (maybe 2.5 too). I don't know much about this either. This may however be doable for those guys who know the Python packaging approach well enough. I don't think eiter of the packages referred at http://trac.osgeo.org/gdal/wiki/DownloadingGdalBinaries would support this feature though. One of the tricks here is which numpy to support, etc. numpy has been pretty good with binary compatibility lately (except for one mistake recently that was corrected) Not sure how this be related to a GDAL binary distribution, as far as I remember numpy can be installed to the Python deployment directly. However, I DON'T want gdal to give me Python -- I use Python for too many other things for that. Yes, adding more runtime environments to a simple GDAL package makes it more heavy weighted. Would also be reasonable to include the Perl environment or a .NET framework installer for example to make it more complete. However, in many cases it's more reasonable to let the application (using the GDAL binaries) decide how to make a proper installer to run their application smoothly. 4) it might be nice if the install location for the utilities got put on the user's PATH -- I don't know how hard that is in an installer -- Windows really sucks in that regard. I don't think it would be beneficial in most cases. This could easily break other applications (using the dll-s with the same names) to fail unexpectedly. I would prefer to keep the applications isolated (setting the environment variables specifically to the process and not the user/system) as much as possible. An installer would better enforce a standard install location. You could also have a custom DOS box with a menu entry that set up the environment for the command line tools (maybe only PATH?), provide an uninstaller, and of course, give the Windows users a nice warm and fuzzy feeling. The 'nice warm and fuzzy feeling' is a good objective indeed, setting up an entry on the start menu instead of a running the batch file directly may also be an advantage. While I'm not sure starting a DOS prompt would validate an installer by it's own, I can see this requirement to be valid in most cases. With regard to Python, an installer could see what Python the user has and install the bindings for that version. Not that I have any idea how to build that! Agreed, but I have no idea either. Inno Setup is a very nice free installer, by the way. I would also add Wix
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
2011/1/5 Daniel Morissette dmorisse...@mapgears.com I agree with you, but it seems that an [OK] button (even if it doesn't do anything) makes Windows users feel so much better. :) Daniel, :-) And sometimes we wonder what a heck is being done behind an OK button on Windows which takes so long (or lasts forever ;-) Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
It may well be that GDAL has too many different use cases to even have a standard install, but... On 1/5/11 1:37 PM, Tamas Szekeres wrote: 2011/1/5 Christopher Barker chris.bar...@noaa.gov 1) It would be nice to have binaries for the latest release front and center at the main GDAL site -- having to poke around to find Tamas's site is not a big deal, but not always obvious. With regards to the comment above, while I'm not sure about the objectives but I don't think the GDAL site would intend to be a hosting provider of various binary packages, Well, many (most?) open source packages have official binaries hosted on its site. It's pretty common to go to a project's site and expect to find a way to download binaries right then and there. 2) A standard install location would be good. As I've messed with this each time, I never know where stuff should go -- maybe installers would help with that This doesn't seem to be decisive requirement to me. It's not a strong requirement, but standard defaults do make things easier for everyone. Or some other folks may prefer installing these files along with their applications or keep such files in separate - project specific - directories. Well, users should certainly be able to do something custom if they want. This is all about use-cases -- if you are building a custom app linked against GDAL, then you probably want to control where you put things. However, if you are interested in the command line tools, and using GDAL via Python or Perl or ??, then it makes it easier to have a standard location. Another issue of an installer may be due to a single product key along with the setup which would prevent from installing multiple versions side by side in the same environment. Surely there are ways to accommodate that? though dll hell is in the lexicon for a reason! 3) If there is a standard install location, then easy_install gdal (or setup.py build, or...) could work for the python bindings. I admit I don't have enough knowledge about the 'magic' tricks related to python-ish way installing applications. That's the thing -- there is no magic here. If you are building a python extension, you need to tell the build system where its dependencies lie. If you are installing a pre-built python extension, then the dependencies need to be in a known place (or maybe on the right PATHS -- Windows is pretty ugly this way). Which is why a standard install location would be a good idea. I might also consider running a custom application with gdal not necessarily be the responsibility of a GDAL package. well, that depends on whether you consider Python bindings a custom application. In any case, I think it helps third party packages to have standard default install locations. Oh for *nix -- this would be easier if we just could just put stuff in /usr/local/... You might also want to install python from a separate installer (either ActivePython, python.org http://python.org whatever) True -- but it is very much a standard for third party packages to provide binaries for the python.org python build. Again, I'm not suggesting that folks should be prevented (or even discouraged) from doing various sorts of custom installs, just that there should be defaults, so that it's clear an easy for a newbie to know what to to to get things to just work. Another option is to have a binary installer for the python bindings that includes gdal and the gdal utilities -- that would be great for users like me, but I don't know how common my use case is. In that case, you'd want to support a few recent pythons versions, the python.org http://python.org binaries: 2.6, 2.7, 3.1 (maybe 2.5 too). I don't know much about this either. This may however be doable for those guys who know the Python packaging approach well enough. It's not that hard (at lest once GDAL is built), but it is work. I don't think eiter of the packages referred at http://trac.osgeo.org/gdal/wiki/DownloadingGdalBinaries would support this feature though. Agreed -- that's my point! One of the tricks here is which numpy to support, etc. numpy has been pretty good with binary compatibility lately (except for one mistake recently that was corrected) Not sure how this be related to a GDAL binary distribution, as far as I remember numpy can be installed to the Python deployment directly. yes, but the Python bindings are built against a particular numpy. That's OK for version so numpy that are binary compatible, but it potentially fragile. Note that with the new extended buffer interface, it should be possible to build GDAL with full numpy support, but not have to compile against numpy. But I think that's only good for 2.7 and 3.* However, I DON'T want gdal to give me Python -- I use Python for too many other things for that. Yes, adding more runtime environments to a simple GDAL
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Folks, Wow, what a lot of discussion. From my perspective, I'm pleased with Tamas' provided binaries as a source for developers who want a GDAL SDK for a particular compiler version and choice of win32/win64. I like Jurgen's idea of building a stock GDAL binaries package with an installer (similar to FWTools) from OSGeo4W packages. Like some others, I'm also confused by the correct way of providing python binaries. I'm inclined to provide an included python in a self installer similar to FWTools, but I'd be pleased if the hardcore Pythonistas can get the magic ways of producing python binaries for any Python version working. Is there someone that wants to take on producing official GDAL binaries from OSGeo4W with a self installer sort of similar to FWTools, likely using the mechanism Jurgen described? I'm interested, but interest hasn't turned into action over quite a few months of interest. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Programmer for Rent ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Specifically regarding Python programmers who primarily or exclusively use Windows: The majority of popular Python packages are delivered to Windows users by one of two mechanisms: 1.A stand-alone installation program (either a .exe or .msi file) that the user just downloads and runs, often built with the Python distutils technology. 2.The easy_install mechanism, in which the user starts a shell and types easy_install x. Most Windows users are not familiar with the UNIX way of doing things, in which the package is compiled and installed from a script. Most Windows users do not even have a C compiler on their machine. As evidence for the popularity of #1, just look at the installation statistics for popular packages such as scipy or numpy. For example, the Python 2.6 .exe installer for scipy 0.8.0 currently has 28417 downloads, while the source code (.tar.gz file) currently has 14403 downloads. As UNIX users will be downloading the source code and not the Windows executable, the 14403 is likely to be a lot of UNIX users and few Windows users. As you can see, the vast majority of Windows scipy users prefer to install it via the GUI installation program rather than the source code. Right now, to install the latest GDAL for Python on Windows, the user has to download a zip file from http://vbkto.dyndns.org/sdk/, drill in to find the Python files, copy them to C:\PythonXX\Lib\site-packages, drill into find the GDAL binaries and related files, copy them to a directory such as C:\GDAL, set the PATH and GDAL environment variables, and so on. And there are no obvious instructions anywhere about how to do this. Python programmers are programmers after all, so they can generally figure that out and accomplish it. But it is not what they are used to doing if they are a Windows programmer. This is why Chris basically says that he has to remember / relearn how to do it every time he upgrades GDAL. It would be really helpful to Windows Python programmers who want to use GDAL-probably a large number of potential GDAL users-for the GDAL team to offer installation via one or both of the mechanisms I mentioned above. Best, Jason From: gdal-dev-boun...@lists.osgeo.org [mailto:gdal-dev-boun...@lists.osgeo.org] On Behalf Of Tamas Szekeres Sent: Wednesday, January 05, 2011 4:37 PM To: Christopher Barker Cc: gdal-dev@lists.osgeo.org Subject: Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0 2011/1/5 Christopher Barker chris.bar...@noaa.gov 1) It would be nice to have binaries for the latest release front and center at the main GDAL site -- having to poke around to find Tamas's site is not a big deal, but not always obvious. Chris, With regards to the comment above, while I'm not sure about the objectives but I don't think the GDAL site would intend to be a hosting provider of various binary packages, the most reasonable thing is to put the references pointing the user to the correct locations which has already been done, see the Downloads section at http://www.gdal.org/ 2) A standard install location would be good. As I've messed with this each time, I never know where stuff should go -- maybe installers would help with that This doesn't seem to be decisive requirement to me. We may also create a definite location in the hard drive to host such files (which can be remembered later). Or some other folks may prefer installing these files along with their applications or keep such files in separate - project specific - directories. We may also have a requirement to deploy and run these applications from portable locations (ie. from CD or a flash drive). Another issue of an installer may be due to a single product key along with the setup which would prevent from installing multiple versions side by side in the same environment. 3) If there is a standard install location, then easy_install gdal (or setup.py build, or...) could work for the python bindings. I admit I don't have enough knowledge about the 'magic' tricks related to python-ish way installing applications. I expect that most 'magic' are implemented by copying the files at certain locations, setting some environment variables or regkey entries. However I might also consider running a custom application with gdal not necessarily be the responsibility of a GDAL package. You might also want to install python from a separate installer (either ActivePython, python.org whatever) and run the application direcly from a command prompt where the environments are set to run the gdal applications properly. Most of these packages provide the required .bat file to setup the environment this way. Another option is to have a binary installer for the python bindings that includes gdal and the gdal utilities -- that would be great for users like me, but I don't know how common my use case is. In that case, you'd want to support a few recent pythons versions, the python.org binaries: 2.6, 2.7, 3.1 (maybe 2.5 too). I don't
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
2011/1/5 Jason Roberts jason.robe...@duke.edu Right now, to install the latest GDAL for Python on Windows, the user has to download a zip file from http://vbkto.dyndns.org/sdk/, drill in to find the Python files, copy them to C:\PythonXX\Lib\site-packages, drill into find the GDAL binaries and related files, copy them to a directory such as C:\GDAL, set the PATH and GDAL environment variables, and so on. And there are no obvious instructions anywhere about how to do this. Python programmers are programmers after all, so they can generally figure that out and accomplish it. But it is not what they are used to doing if they are a Windows programmer. This is why Chris basically says that he has to remember / relearn how to do it every time he upgrades GDAL. It would be really helpful to Windows Python programmers who want to use GDAL—probably a large number of potential GDAL users—for the GDAL team to offer installation via one or both of the mechanisms I mentioned above. Jason, Maybe this is due to my lack of knowledge in Python, but why is it a requirement to place the files to such specific locations to run a python app with gdal on Windows? As far as I remember setting up the following environment variables properly (in a command prompt) does the right thing from arbitrary locations (at least when running the autotest suite for GDAL): PROJ_LIB GDAL_DRIVER_PATH GDAL_DATA PYTHONPATH PATH Certainly one should find out the location of the python.exe which might also be included in PATH for convenience. Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Chris, Good points below, but having the compiled gdal binaries (and the binaries of the dependent libraries) in hand, which is the right way to install those files on Windows? (Assuming we don't provide python.exe and the related files in the package)? I mean which install actions should be done in detail? Best regards, Tamas 2011/1/5 Christopher Barker chris.bar...@noaa.gov It may well be that GDAL has too many different use cases to even have a standard install, but... On 1/5/11 1:37 PM, Tamas Szekeres wrote: 2011/1/5 Christopher Barker chris.bar...@noaa.gov 1) It would be nice to have binaries for the latest release front and center at the main GDAL site -- having to poke around to find Tamas's site is not a big deal, but not always obvious. With regards to the comment above, while I'm not sure about the objectives but I don't think the GDAL site would intend to be a hosting provider of various binary packages, Well, many (most?) open source packages have official binaries hosted on its site. It's pretty common to go to a project's site and expect to find a way to download binaries right then and there. 2) A standard install location would be good. As I've messed with this each time, I never know where stuff should go -- maybe installers would help with that This doesn't seem to be decisive requirement to me. It's not a strong requirement, but standard defaults do make things easier for everyone. Or some other folks may prefer installing these files along with their applications or keep such files in separate - project specific - directories. Well, users should certainly be able to do something custom if they want. This is all about use-cases -- if you are building a custom app linked against GDAL, then you probably want to control where you put things. However, if you are interested in the command line tools, and using GDAL via Python or Perl or ??, then it makes it easier to have a standard location. Another issue of an installer may be due to a single product key along with the setup which would prevent from installing multiple versions side by side in the same environment. Surely there are ways to accommodate that? though dll hell is in the lexicon for a reason! 3) If there is a standard install location, then easy_install gdal (or setup.py build, or...) could work for the python bindings. I admit I don't have enough knowledge about the 'magic' tricks related to python-ish way installing applications. That's the thing -- there is no magic here. If you are building a python extension, you need to tell the build system where its dependencies lie. If you are installing a pre-built python extension, then the dependencies need to be in a known place (or maybe on the right PATHS -- Windows is pretty ugly this way). Which is why a standard install location would be a good idea. I might also consider running a custom application with gdal not necessarily be the responsibility of a GDAL package. well, that depends on whether you consider Python bindings a custom application. In any case, I think it helps third party packages to have standard default install locations. Oh for *nix -- this would be easier if we just could just put stuff in /usr/local/... You might also want to install python from a separate installer (either ActivePython, python.org http://python.org whatever) True -- but it is very much a standard for third party packages to provide binaries for the python.org python build. Again, I'm not suggesting that folks should be prevented (or even discouraged) from doing various sorts of custom installs, just that there should be defaults, so that it's clear an easy for a newbie to know what to to to get things to just work. Another option is to have a binary installer for the python bindings that includes gdal and the gdal utilities -- that would be great for users like me, but I don't know how common my use case is. In that case, you'd want to support a few recent pythons versions, the python.org http://python.org binaries: 2.6, 2.7, 3.1 (maybe 2.5 too). I don't know much about this either. This may however be doable for those guys who know the Python packaging approach well enough. It's not that hard (at lest once GDAL is built), but it is work. I don't think eiter of the packages referred at http://trac.osgeo.org/gdal/wiki/DownloadingGdalBinaries would support this feature though. Agreed -- that's my point! One of the tricks here is which numpy to support, etc. numpy has been pretty good with binary compatibility lately (except for one mistake recently that was corrected) Not sure how this be related to a GDAL binary distribution, as far as I remember numpy can be installed to the Python deployment directly. yes, but the Python bindings are built against a particular numpy. That's OK for version so
RE: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Tamas, I have a decent amount of experience programming with Python (6 years, develop my own extension modules, etc) and I have never seen a package that recommends you install it to a directory of your choice and then modify the PYTHONPATH environment variable. The standard installation location is the site-packages directory. See http://www.python.org/dev/peps/pep-0250/ for some background on that. The GDAL team certainly could publish instructions that said copy the release-1500-dev\gdal\swig\python\*.py and the osgeo subdirectory directory to a location of your choice and modify set the PYTHONPATH environment variable to include that location but it would be very unusual. GDAL would likely be the only package that the user encounters with those instructions. The PYTHONPATH environment variable would likely not even be defined already. (Although if they have ArcGIS, it would be, because ArcGIS includes some Python extensions that live in the ArcGIS installation dir.) So, in sum, you could make PYTHONPATH your official solution but it would be weird. The more traditional route would be to modify the GDAL build scripts to invoke the existing setup.py with the arguments bdist --formats=wininst to build a Windows installation package (see http://docs.python.org/distutils/builtdist.html). Ideally, it would also include a post-install script that could install the GDAL binaries somewhere (such as a subdir inside \PythonXX\Lib\site-packages\gdal) and set PATH, PROJ_LIB, GDAL_DRIVER_PATH, and GDAL_DATA to the correct things. I could potentially help you develop that. It looks like most of what is needed is already there. Jason From: Tamas Szekeres [mailto:szeker...@gmail.com] Sent: Wednesday, January 05, 2011 6:12 PM To: Jason Roberts Cc: gdal-dev@lists.osgeo.org; Christopher Barker Subject: Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0 2011/1/5 Jason Roberts jason.robe...@duke.edu Right now, to install the latest GDAL for Python on Windows, the user has to download a zip file from http://vbkto.dyndns.org/sdk/, drill in to find the Python files, copy them to C:\PythonXX\Lib\site-packages, drill into find the GDAL binaries and related files, copy them to a directory such as C:\GDAL, set the PATH and GDAL environment variables, and so on. And there are no obvious instructions anywhere about how to do this. Python programmers are programmers after all, so they can generally figure that out and accomplish it. But it is not what they are used to doing if they are a Windows programmer. This is why Chris basically says that he has to remember / relearn how to do it every time he upgrades GDAL. It would be really helpful to Windows Python programmers who want to use GDAL-probably a large number of potential GDAL users-for the GDAL team to offer installation via one or both of the mechanisms I mentioned above. Jason, Maybe this is due to my lack of knowledge in Python, but why is it a requirement to place the files to such specific locations to run a python app with gdal on Windows? As far as I remember setting up the following environment variables properly (in a command prompt) does the right thing from arbitrary locations (at least when running the autotest suite for GDAL): PROJ_LIB GDAL_DRIVER_PATH GDAL_DATA PYTHONPATH PATH Certainly one should find out the location of the python.exe which might also be included in PATH for convenience. Best regards, Tamas ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
On 1/5/11 2:32 PM, Frank Warmerdam wrote: From my perspective, I'm pleased with Tamas' provided binaries as a source for developers who want a GDAL SDK for a particular compiler version and choice of win32/win64. yes, those are great! Like some others, I'm also confused by the correct way of providing python binaries. I'm inclined to provide an included python in a self installer similar to FWTools, This is really nice for folks that aren't using Python for other things, but if GDAL is only part of your Python tools stack, not so great. but I'd be pleased if the hardcore Pythonistas can get the magic ways of producing python binaries for any Python version working. I think it's possible, though I'm not sure about any! maybe the most common Is there someone that wants to take on producing official GDAL binaries from OSGeo4W I'm confused about what from OSGeo4w means, and I don't know that I can take it on, but I've spent enough time on this thread already, I should help out! On 1/5/11 2:43 PM, Jason Roberts wrote: The majority of popular Python packages are delivered to Windows users by one of two mechanisms: 1.A stand-alone installation program (either a .exe or .msi file) that the user just downloads and runs, often built with the Python “distutils” technology. 2.The easy_install mechanism, in which the user starts a shell and types “easy_install x”. absolutely true. It's also not that hard to do one of those, once you've got the other. Once you've got the .msi building, building a binary egg is pretty easy, and then you just have to make sure pypi has the right links in place so that people get it with easy_install. In short, we have one problem to solve, not two. Right now, to install the latest GDAL for Python on Windows, the user has to download a zip file from http://vbkto.dyndns.org/sdk/, drill in to find the Python files, copy them to C:\PythonXX\Lib\site-packages, drill into find the GDAL binaries and related files, copy them to a directory such as C:\GDAL, set the PATH and GDAL environment variables, and so on. And there are no obvious instructions anywhere about how to do this. Python programmers are programmers after all, so they can generally figure that out and accomplish it. But it is not what they are used to doing if they are a Windows programmer. This is why Chris basically says that he has to remember / relearn how to do it every time he upgrades GDAL. Good summary -- yes, that's pretty much it. Which is why I'm saying that if we had a standard way that GDAL binaries got installed, it would be a lot easier. Then it would be: 1) install GDAL binaries 2) either: a) download the source for the python bindings and setup.py build or b) run easy_install gdal (which would download the source and build the binary) It would be really helpful to Windows Python programmers who want to use GDAL—probably a large number of potential GDAL users—for the GDAL team to offer installation via one or both of the mechanisms I mentioned above. yup. On 1/5/11 3:12 PM, Tamas Szekeres wrote: Maybe this is due to my lack of knowledge in Python, but why is it a requirement to place the files to such specific locations to run a python app with gdal on Windows? well, it's not the only way to do it, but it is the most common way. As far as I remember setting up the following environment variables properly (in a command prompt) does the right thing from arbitrary locations (at least when running the autotest suite for GDAL): PROJ_LIB GDAL_DRIVER_PATH GDAL_DATA PYTHONPATH PATH well -- first of all, setting environment variables on Windows is a PITA, second of all, is setting all that up easier than what Jason described?, and third, PYTHONPATH is a problem, particularly if you have more than one version of python installed. So I think Jason's way is a better option, but it really would be nice to do an installer. so what to do? This is all complicated because GDAL is many things: - a lib that can be used to build your own C/C++, etc. apps - a set of command line utilities - a python package - a package for various other languages - a set up utilities written in Python, also. Being a Pythonista, I'll focus on the Python bits Question 1: Do we want: a) a binary package for the python bindings that is completely stand-alone -- i.e. has its own copy of everything GDAL that it needs, and does not share the dlls with anythign else. b) (a) but sharing the dlls with other things that may need gdal? (other language bindings, mapserver, what have you) c) (a) or (b) plus the python command line tools d) (a), (b), or (c) plus the compiled command line tools. As I write this, I realize that maybe using or following OSGeo4w is the way to go -- put the gdal binaries wherever OSGeo4w puts them, and building the python bindings against that. Essentially letting OSGeo4w establish the standard. To Tamas: I wonder if there is some confusion
Re: Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Hi Frank, On Wed, 05. Jan 2011 at 17:32:42 -0500, Frank Warmerdam wrote: Is there someone that wants to take on producing official GDAL binaries from OSGeo4W with a self installer sort of similar to FWTools, likely using the mechanism Jurgen described? I'm interested, but interest hasn't turned into action over quite a few months of interest. A starting point could be [1]. It's result is [2]. Jürgen [1] http://qgis.org/~jef/creatensis.tar.bz2 [2] http://qgis.org/~jef/GDAL-OSGeo4W-1.7.3-Setup.exe -- Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-20 Dipl.-Inf. (FH) Rheinstraße 13Fax. +49-4931-918175-50 Software Engineer D-26506 Norden http://www.norbit.de -- norBIT Gesellschaft fuer Unternehmensberatung und Informationssysteme mbH Rheinstrasse 13, 26506 Norden GF: Jelto Buurman, HR: Amtsgericht Emden, HRB 5502 ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Fwd: Re: [gdal-dev] FWTools and GDAL 1.7.0
Hi Frank I personally will be happy to see FWTOOLS updated at least for major Gdal releases. I find it to be a much simpler way to distribute Gdal to my end users. I agree that OSGeo4W is more complete, but I think that for many users the simplicity of FWTOOLS wins. Thanks Yehiyam Livneh From: Frank Warmerdam Subject: Re: [gdal-dev] FWTools and GDAL 1.7.0 Date: 04 ינואר 2011 05:11 On 11-01-03 06:40 PM, Kyle Shannon wrote: Hello all, I was working on #3890 and the ticket issuer reported gdalinfo version as: gdalinfo --version GDAL 1.7.0b2, FWTools 2.4.7, released 2010/01/19 I downloaded FWTools 2.4.7 for windows and verified the version. I may be confused with the GDAL version, but assuming b2 is beta 2, wasn't this version was retracted? Does anyone know if beta 2 was released before the commit that caused the HFA issue? Just wondering if the FWTools should be upgraded to 1.7.x. Thanks. Kyle, FWTools was built at random intervals from GDAL trunk. It appears that last release was around the 1.7 release point. I haven't really tried to keep producing new FWTools releases over the last year and I'm not sure I've retained enough of the materials for it to produce a new one. I might revisit this at some point. In the meantime I mostly try to point people to OSGeo4W now. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush | Geospatial Programmer for Rent ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ** This message (including any attachments) issued by RAFAEL- ADVANCED DEFENSE SYSTEMS LTD. (hereinafter RAFAEL) contains confidential information intended for a specific individual and purpose, may constitute information that is privileged or confidential or otherwise protected from disclosure. If you are not the intended recipient, you should contact us immediately and thereafter delete this message from your system. You are hereby notified that any disclosure, copying, dissemination, distribution or forwarding of this message, or the taking of any action based on it, is strictly prohibited. If you have received this e-mail in error, please notify us immediately by e-mail mailto:law...@rafael.co.il and completely delete or destroy any and all electronic or other copies of the original message and any attachments thereof. ** ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev