Ozy,
The interesting info is that your input image is JPEG2000 compressed.
This explains why you were able to read a scanline oriented NITF with
blockwidth > 9999. My guess would be that the leak is in the JPEG2000
driver in question, so this may be more a problem on the reading part
than on the writing part. You can check that by running : gdalinfo
-checksum NITF_IM:0:input.ntf. If you see the memory increasing again
and again, there's definitely a problem. In case you have GDAL
configured with several JPEG2000 drivers, you'll have to find which one
is used : JP2KAK (Kakadu based), JP2ECW (ECW SDK based), JPEG2000
(Jasper based, but I doubt you're using it with such a big dataset),
JP2MRSID. Normally, they are selected in the order I've described
(JP2KAK first, etc). As you're on Linux, it might be interesting that
you run valgrind to see if it reports leaks. As it might very slow on
such a big dataset, you could try translating just a smaller window of
your input dataset, like
valgrind --leak-check=full gdal_translate NITF_IM:0:input.ntf output.tif
-srcwin 0 0 37504 128
I've selected TIF as output format as it shouldn't matter if you confirm
that the problem is in the reading part. As far as the window size is
concerned, it's difficult to guess which value will show the leak.
Filing a ticket with your findings on GDAL Trac might be appropriate.
It might be good trying with GDAL trunk first though, in case the leak
might have been fixed since 1.6.2. The beta2 source zip are to be found
here : http://download.osgeo.org/gdal/gdal-1.7.0b2.tar.gz
Best regards,
Even
ozy sjahputera a écrit :
Hi Even,
yes, I tried:
gdal_translate -of "NITF" -co "ICORDS=G" -co "BLOCKXSIZE=128" -co
"BLOCKYSIZE=128" NITF_IM:0:input.ntf output.ntf
I monitored the memory use using top and it was steadily increasing
till it reached 98.4% (I have 8GB of RAM and 140 GB of local disk for
swap etc.) before the node died (not just the program, but the whole
system just stopped responding).
My GDAL version is 1.6.2.
gdalinfo on this image shows the raster size of (37504, 98772) and
Block=37504x1.
The image is compressed using JPEG2000 option and contains two
subdatasets (data and cloud data ~ I used only the data for
gdal_translate test).
Band info from gdalinfo:
Band 1 Block=37504x1 Type=UInt16, ColorInterp=Gray
Ozy
On Tue, Jan 12, 2010 at 5:38 PM, Even Rouault
<even.roua...@mines-paris.org <mailto:even.roua...@mines-paris.org>>
wrote:
Ozy,
Did you try with gdal_translate -of NITF src.tif output.tif -co
BLOCKSIZE=128 ? Does it give similar results ?
I'm a bit surprised that you even managed to read a 40Kx100K large
NITF
file organized as scanlines. There was a limit until very recently
that
prevented to read blocks whose one dimension was bigger than 9999.
This
was fixed recently in trunk ( see ticket
http://trac.osgeo.org/gdal/ticket/3263 ) and branches/1.6, but it has
not yet been released to an officially released version. So which GDAL
version are you using ?
Does the output of gdalinfo on your scanline oriented input NITF gives
something like :
Band 1 Block=40000x1 Type=Byte, ColorInterp=Gray
Is your input NITF compressed or uncompressed ?
Anyway, with latest trunk, I've simulated creating a similarly large
NITF image with the following python snippet :
import gdal
ds = gdal.GetDriverByName('NITF').Create('scanline.ntf', 40000,
100000)
ds = None
and then creating the tiled NITF :
gdal_translate -of NITF scanline.ntf tiled.ntf -co BLOCKSIZE=128
The memory consumption is very reasonnable (less than 50 MB : the
default block cache size of 40 MB + temporary buffers ), so I'm not
clear why you would have a problem of increasing memory use.
ozy sjahputera a écrit :
> I was trying to make a copy of a very large NITF image (about
40Kx100K
> pixels) using GDALDriver::CreateCopy(). The new file was set to have
> different block-size (input was a scanline image, output is to
have a
> 128x128 blocksize). The program keeps getting killed by the system
> (Linux). I monitor the memory use of the program as it was executing
> CreateCopy and the memory use was steadily increasing as the
progress
> indicator from CreateCopy was moving forward.
>
> Why does CreateCopy() use so much memory? I have not perused the
> source code of CreateCopy() yet, but I am guessing it employs
> RasterIO() to perform the read/write?
>
> I was trying different sizes for GDAL cache from 64MB, 256MB,
512MB,
> 1GB, and 2GB. The program got killed in all these cache sizes. In
> fact, my Linux box became unresponsive when I set
GDALSetCacheMax() to
> 64MB.
>
> Thank you.
> Ozy
>
>
>
------------------------------------------------------------------------
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org <mailto:gdal-dev@lists.osgeo.org>
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev