Re: [gdal-dev] Memory fragmentation and memory consumption when warping with GCPs
Joaquim Luis wrote: After that I wrote a MEX version of gdaltransform to be able to reproject points using GCPs. This MEX works but it revealed serious problems. At least under Matlab. Matlab requires not only LOTS of memory but also that the available memory is continuous. For example, if one needs a variable of 100 Mb the system needs to have a chunk of free memory not less than 100 Mb. Two chunks of 99 and 1 Mb are not enough as we'll have an out memory error. Now, I just realize that when I call a MEX that links against gdal.dll the memory immediately becomes highly fragmented. As an example of my current case, I had the largest chunk of free memory of 1070 Mb and after loading gdal.dll the largest chunk become 540 Mb Hello, I follow up on the memory fragmentation issue. I found that the fragmentation is caused by the ECW driver. If compiled without that driver, GDAL doesn't provoke any sensible fragmentation. Joaquim Luis ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Symbian Porting Issues
Le Monday 05 January 2009 18:01:31 Mark Wilcox, vous avez écrit : > Hi, > > First off, I must congratulate Frank and all the GDAL contributors on > producing a fantastically portable library! With just 2 days of effort > starting from basically zero knowledge of GDAL I've got the core running > and many of the raster formats building for Symbian OS. I've even tested > JPEGs and GeoTIFFs with a simple viewer application in the Symbian > emulator. > Good to hear! > > Python is available on Symbian and Nokia have just released a new version > (upgrading to 2.5.2 from 2.2) so I'll look and see if it's feasible to > build and run the autotest stuff too as soon as I have time - is there much > C/C++ test code other than that though? There is some C/C++ code that lies in autotest/cpp directory, but it is not much maintened and used AFAIK. I would consider the python autotests as the reference test system for GDAL. They are a test for the python bindings of course, but also for the core functionnality and the drivers. Last time I measured, their coverage rate was 42.8% of the compiled code of my build (getting an extra .1% requires a non trivial amount of work ;-)). For GDAL and OGR core, port library, and tested drivers, it is typically between 60% and 80%. Non-tested drivers make the overall figure drop. > > 1) For some reason inflateCopy() is missing from the Symbian zlib port, > even though it claims to be version 1.2.3 (existing library shipped in > device firmware, not the internal copy in GDAL) at the moment I'm just > excluding cpl_vsil_gzip.cpp from my build. I'll try to get Nokia/Symbian > to fix this and include the function in future releases. I've seen some > archive post that suggests maybe the same functionality could have been > achieved without this dependency to support older versions of zlib > though... is it worth reconsidering that option for Symbian support or is > the gzip support not so commonly used? ./configure should normally default to internal zlib if it doesn't find inflateCopy symbol. But you could also discard cpl_vsil_gzip.cpp as it is a new functionnality to GDAL 1.6 and is somehow optionnal. (People can always uncompress their data at hand before running GDAL on them) As far as not requiring inflateCopy() for the cpl_vsil_gzip functionnality, that could have been achieved, but the functionnality would not operate in a very efficient way then, so I've not strong motivation to support that. Best regards, Even ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Symbian Porting Issues
Mark Wilcox wrote: 1) For some reason inflateCopy() is missing from the Symbian zlib port, even though it claims to be version 1.2.3 (existing library shipped in device firmware, not the internal copy in GDAL) at the moment I'm just excluding cpl_vsil_gzip.cpp from my build. I'll try to get Nokia/Symbian to fix this and include the function in future releases. I've seen some archive post that suggests maybe the same functionality could have been achieved without this dependency to support older versions of zlib though... is it worth reconsidering that option for Symbian support or is the gzip support not so commonly used? Mark, I have no strong opinion on this. I presume things work fine if the internal gdal/frmts/zlib is used, right? 2) /frmts/hfa/addtiffo_src/rawblockedimage.cpp contains: #ifndef _WIN32 # include #endif should this be #ifndef WIN32 or possibly better yet #ifdef HAVE_UNISTD_H? Currently the build will fail for a Symbian emulator because _WIN32 is defined but you're effectively cross-compiling to an odd hybrid environment. The addtiffo_src stuff is not used except by img2tif. It should not be included in normal GDAL builds. 3) The zlib library also uses a WIN32 define for portablility, this caused clashes in several places which I've temporarily fixed by changing the order of the includes (so that zlib.h comes after any gdal headers). This is very unsatisfactory but I'm not sure what the real solution is. One option would be to get the Symbian port of zlib fixed so it doesn't export this definition to the world in it's public header, another would be to change the cpl_port.h to undef WIN32 before re-checking for itself. I thought possibly the latter since zlib isn't updated so often? I'm not sure I understand this issue. If zlib.h defines WIN32 on a platform that GDAL does not really consider WIN32 then I suppose we ought to correct the definition right after including zlib.h everywhere it is included (hopefully not too many places). 4) There are a couple of symbol clashes at link time between /frmts/hfa/geoextra.c & /frmts/gtiff/libgeotiff/geo_extra.c. I assume that one of them is not normally built? I've excluded the hfa version from the build for now but is there a case for removing one permanently, or moving the functions to a common location shared by both drivers? The implementations are slightly different. gdal/frmts/hfa/geoextra.c is not normally built into GDAL. It is present only for use with the img2tif utility. I will take the liberty of clearing away the img2tiff stuff which is no longer well maintained, and is not really used. 5) Symbian doesn't have an lfind() implementation at the moment. I've borrowed the one from the full version of libtiff's port directory but wondered if there might be a case for including a version more permanently in GDAL, either with the internal copy of libtiff or perhaps in the porting layer? I'm ambivelent on this. I think it is sufficient to conditionally compile in an lfind() implementation for the Symbian and I wouldn't mind this living somewhere appropriate in the GDAL tree (perhaps gdal/port). Congratulations on the Symbian work. Feel free to submit tickets with suggested changes in support of the platform. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Programmer for Rent ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Symbian Porting Issues
Hi, First off, I must congratulate Frank and all the GDAL contributors on producing a fantastically portable library! With just 2 days of effort starting from basically zero knowledge of GDAL I've got the core running and many of the raster formats building for Symbian OS. I've even tested JPEGs and GeoTIFFs with a simple viewer application in the Symbian emulator. A lot of tidying and testing is still required plus the vector formats for OGR (although the core OGR functionality is building already) and maybe the utilities and python bindings. I took a look at the code and decided the fastest approach would probably be to try to build everything directory by directory, excluding things with external dependencies and non-trivial errors for future examination. As such I'm probably building some things I shouldn't be but I'll look at a proper configuration once I've got my new laptop setup to dual-boot Linux with Vista (I'm much more familiar with the Linux build environment, despite being forced to use Windows for Symbian stuff at the moment). Python is available on Symbian and Nokia have just released a new version (upgrading to 2.5.2 from 2.2) so I'll look and see if it's feasible to build and run the autotest stuff too as soon as I have time - is there much C/C++ test code other than that though? Most of the issues I came across were related to the Nokia compiler (ex-Metrowerks) for the windows emulator environment being more fussy about implicit casts than anything else GDAL is built with. However, in the process of porting I've come across some issues that may be worth further investigation by those more knowledgeable about the code base: 1) For some reason inflateCopy() is missing from the Symbian zlib port, even though it claims to be version 1.2.3 (existing library shipped in device firmware, not the internal copy in GDAL) at the moment I'm just excluding cpl_vsil_gzip.cpp from my build. I'll try to get Nokia/Symbian to fix this and include the function in future releases. I've seen some archive post that suggests maybe the same functionality could have been achieved without this dependency to support older versions of zlib though... is it worth reconsidering that option for Symbian support or is the gzip support not so commonly used? 2) /frmts/hfa/addtiffo_src/rawblockedimage.cpp contains: #ifndef _WIN32 # include #endif should this be #ifndef WIN32 or possibly better yet #ifdef HAVE_UNISTD_H? Currently the build will fail for a Symbian emulator because _WIN32 is defined but you're effectively cross-compiling to an odd hybrid environment. 3) The zlib library also uses a WIN32 define for portablility, this caused clashes in several places which I've temporarily fixed by changing the order of the includes (so that zlib.h comes after any gdal headers). This is very unsatisfactory but I'm not sure what the real solution is. One option would be to get the Symbian port of zlib fixed so it doesn't export this definition to the world in it's public header, another would be to change the cpl_port.h to undef WIN32 before re-checking for itself. I thought possibly the latter since zlib isn't updated so often? 4) There are a couple of symbol clashes at link time between /frmts/hfa/geoextra.c & /frmts/gtiff/libgeotiff/geo_extra.c. I assume that one of them is not normally built? I've excluded the hfa version from the build for now but is there a case for removing one permanently, or moving the functions to a common location shared by both drivers? The implementations are slightly different. 5) Symbian doesn't have an lfind() implementation at the moment. I've borrowed the one from the full version of libtiff's port directory but wondered if there might be a case for including a version more permanently in GDAL, either with the internal copy of libtiff or perhaps in the porting layer? Does anyone have any comments or suggestions on any of the above issues? Does anyone have any experience using the autotest in a cross-compilation environment? (Maybe I need to talk to Mateusz about that?) Apologies for the long mail. Best regards, Mark ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
