Re: [gdal-dev] Memory fragmentation and memory consumption when warping with GCPs
Joaquim Luis wrote: After that I wrote a MEX version of gdaltransform to be able to reproject points using GCPs. This MEX works but it revealed serious problems. At least under Matlab. Matlab requires not only LOTS of memory but also that the available memory is continuous. For example, if one needs a variable of 100 Mb the system needs to have a chunk of free memory not less than 100 Mb. Two chunks of 99 and 1 Mb are not enough as we'll have an out memory error. Now, I just realize that when I call a MEX that links against gdal.dll the memory immediately becomes highly fragmented. As an example of my current case, I had the largest chunk of free memory of 1070 Mb and after loading gdal.dll the largest chunk become 540 Mb Hello, I follow up on the memory fragmentation issue. I found that the fragmentation is caused by the ECW driver. If compiled without that driver, GDAL doesn't provoke any sensible fragmentation. Joaquim Luis ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Memory fragmentation and memory consumption when warping with GCPs
Does this aspect really have anything to do with the number of GCPs? Are you using the GDAL warp algorithm? Oh yes, with 2500 GCPs I can do the warping. And yes I'm using the GDAL warp algorithm (the mex is more or less a copy of gdaltransform) Sorry for being so dumb. The GCP warping memory problem was due to a stupid error of over allocation - nGCPCount*nGCPCount instead of the needed nGCPCount It works fine now. But the memory fragmentation issue is another story, and that still remains. Joaquim Luis ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Memory fragmentation and memory consumption when warping with GCPs
Joaquim, OK. Well, generally speaking GDAL should allocate relatively little memory just on loading the DLL (and perhaps calling GDALAllRegister()). So I don't know why you are seeing the problem you see. Frank, It's ok. I was not expecting an easy answer but I think this issue worth reporting. Anyway, the fragmentation (note that there is not a total free memory reduction) is probably originated by all the gdal.dll dependencies that one can see with the Dependency Walker being loaded by the gdal.dll Does this aspect really have anything to do with the number of GCPs? Are you using the GDAL warp algorithm? Oh yes, with 2500 GCPs I can do the warping. And yes I'm using the GDAL warp algorithm (the mex is more or less a copy of gdaltransform) Thanks Joaquim ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Memory fragmentation and memory consumption when warping with GCPs
Joaquim Luis wrote: Frank, Maybe I was not clear enough, but the memory segmentation has nothing to do with the GCP warping. It results from loading the gdal.dll For example, on a fresh start if I call (one other MEX for reading) gdalread without arguments, it prints the usage on screen. After this, which did not perform any calculus, the memory is already fragmented. Joaquim, OK. Well, generally speaking GDAL should allocate relatively little memory just on loading the DLL (and perhaps calling GDALAllRegister()). So I don't know why you are seeing the problem you see. The issue with the GCPs is that with a largest chunk of ~500 Mb I'm not able to warp with 5000 GCPs. Does this aspect really have anything to do with the number of GCPs? Are you using the GDAL warp algorithm? There are things you can control about the amount of memory it tries to use that might be relavent. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, [EMAIL PROTECTED] light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Programmer for Rent ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Memory fragmentation and memory consumption when warping with GCPs
Frank Warmerdam wrote: Joaquim Luis wrote: So one question is, why warping with GCPs takes such large amount of memory (apparently a chunk of 500 Mb was not enough to process 5000 GCPs)? Joaquim, Is the memory fragmentation really related to the use of many GCPs? Does the same problem occur with 4 GCPs? Frank, Maybe I was not clear enough, but the memory segmentation has nothing to do with the GCP warping. It results from loading the gdal.dll For example, on a fresh start if I call (one other MEX for reading) gdalread without arguments, it prints the usage on screen. After this, which did not perform any calculus, the memory is already fragmented. The issue with the GCPs is that with a largest chunk of ~500 Mb I'm not able to warp with 5000 GCPs. Joaquim I also tried with the command line GDAL gdaltransform and here I had no problems. So I'm a bit puzzled. Do all programs depend on the memory fragmentation issue, or is it just a Matlab limitation? (Ah, Windows XP here) As a side note, I also have MEXs that link with the OpenCV library, but those do not sensibly fragment the memory. It is possible that OpenCV uses a private heap thereby minimising the amount of fragmentation in the "master heap". It might also be that OpenEV just doesn't use that much memory. I suspect you are seeing some heap fragmentation and consumption due to GDAL's block caching mechanism but it is really hard to know. You can influence the amount of memory available for the block cache using GDALSetCacheMax(). I'm not in the habit of monitoring heap fragmentation on Win32. I will say that GDAL normally uses malloc() and free() for memory allocation so that may have some implications for heap effects. Best regards, ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Memory fragmentation and memory consumption when warping with GCPs
Joaquim Luis wrote: So one question is, why warping with GCPs takes such large amount of memory (apparently a chunk of 500 Mb was not enough to process 5000 GCPs)? Joaquim, Is the memory fragmentation really related to the use of many GCPs? Does the same problem occur with 4 GCPs? I also tried with the command line GDAL gdaltransform and here I had no problems. So I'm a bit puzzled. Do all programs depend on the memory fragmentation issue, or is it just a Matlab limitation? (Ah, Windows XP here) As a side note, I also have MEXs that link with the OpenCV library, but those do not sensibly fragment the memory. It is possible that OpenCV uses a private heap thereby minimising the amount of fragmentation in the "master heap". It might also be that OpenEV just doesn't use that much memory. I suspect you are seeing some heap fragmentation and consumption due to GDAL's block caching mechanism but it is really hard to know. You can influence the amount of memory available for the block cache using GDALSetCacheMax(). I'm not in the habit of monitoring heap fragmentation on Win32. I will say that GDAL normally uses malloc() and free() for memory allocation so that may have some implications for heap effects. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, [EMAIL PROTECTED] light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Programmer for Rent ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Memory fragmentation and memory consumption when warping with GCPs
Hi, I managed to solve my problem with the GCPs in MEM driver (and sorry if I bothered some with that thread), which turned up to be due to a bad initialization of the dataset geotransform. After that I wrote a MEX version of gdaltransform to be able to reproject points using GCPs. This MEX works but it revealed serious problems. At least under Matlab. Matlab requires not only LOTS of memory but also that the available memory is continuous. For example, if one needs a variable of 100 Mb the system needs to have a chunk of free memory not less than 100 Mb. Two chunks of 99 and 1 Mb are not enough as we'll have an out memory error. Now, I just realize that when I call a MEX that links against gdal.dll the memory immediately becomes highly fragmented. As an example of my current case, I had the largest chunk of free memory of 1070 Mb and after loading gdal.dll the largest chunk become 540 Mb Eventually due to this memory fragmentation, I could not reproject a file having a large number of GCPs (> 5000 GCPs). By decreasing the number of GCPs to half I was able to perform the operation. So one question is, why warping with GCPs takes such large amount of memory (apparently a chunk of 500 Mb was not enough to process 5000 GCPs)? I also tried with the command line GDAL gdaltransform and here I had no problems. So I'm a bit puzzled. Do all programs depend on the memory fragmentation issue, or is it just a Matlab limitation? (Ah, Windows XP here) As a side note, I also have MEXs that link with the OpenCV library, but those do not sensibly fragment the memory. Thanks Joaquim Luis ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
