Update: after more than 20 minutes of being non-responsive, the OS finally regained functionality and promptly killed gdal_translate after about 80% into the process.
On Wed, Jan 13, 2010 at 11:14 AM, ozy sjahputera <sjahpute...@gmail.com>wrote: > Hi Even, > > yes, I tried: > gdal_translate -of "NITF" -co "ICORDS=G" -co "BLOCKXSIZE=128" -co > "BLOCKYSIZE=128" NITF_IM:0:input.ntf output.ntf > > I monitored the memory use using top and it was steadily increasing till it > reached 98.4% (I have 8GB of RAM and 140 GB of local disk for swap etc.) > before the node died (not just the program, but the whole system just > stopped responding). > > My GDAL version is 1.6.2. > > gdalinfo on this image shows the raster size of (37504, 98772) and > Block=37504x1. > The image is compressed using JPEG2000 option and contains two subdatasets > (data and cloud data ~ I used only the data for gdal_translate test). > > Band info from gdalinfo: > Band 1 Block=37504x1 Type=UInt16, ColorInterp=Gray > > Ozy > > > On Tue, Jan 12, 2010 at 5:38 PM, Even Rouault < > even.roua...@mines-paris.org> wrote: > >> Ozy, >> >> Did you try with gdal_translate -of NITF src.tif output.tif -co >> BLOCKSIZE=128 ? Does it give similar results ? >> >> I'm a bit surprised that you even managed to read a 40Kx100K large NITF >> file organized as scanlines. There was a limit until very recently that >> prevented to read blocks whose one dimension was bigger than 9999. This >> was fixed recently in trunk ( see ticket >> http://trac.osgeo.org/gdal/ticket/3263 ) and branches/1.6, but it has >> not yet been released to an officially released version. So which GDAL >> version are you using ? >> >> Does the output of gdalinfo on your scanline oriented input NITF gives >> something like : >> Band 1 Block=40000x1 Type=Byte, ColorInterp=Gray >> >> Is your input NITF compressed or uncompressed ? >> >> Anyway, with latest trunk, I've simulated creating a similarly large >> NITF image with the following python snippet : >> >> import gdal >> ds = gdal.GetDriverByName('NITF').Create('scanline.ntf', 40000, 100000) >> ds = None >> >> and then creating the tiled NITF : >> >> gdal_translate -of NITF scanline.ntf tiled.ntf -co BLOCKSIZE=128 >> >> The memory consumption is very reasonnable (less than 50 MB : the >> default block cache size of 40 MB + temporary buffers ), so I'm not >> clear why you would have a problem of increasing memory use. >> >> ozy sjahputera a écrit : >> > I was trying to make a copy of a very large NITF image (about 40Kx100K >> > pixels) using GDALDriver::CreateCopy(). The new file was set to have >> > different block-size (input was a scanline image, output is to have a >> > 128x128 blocksize). The program keeps getting killed by the system >> > (Linux). I monitor the memory use of the program as it was executing >> > CreateCopy and the memory use was steadily increasing as the progress >> > indicator from CreateCopy was moving forward. >> > >> > Why does CreateCopy() use so much memory? I have not perused the >> > source code of CreateCopy() yet, but I am guessing it employs >> > RasterIO() to perform the read/write? >> > >> > I was trying different sizes for GDAL cache from 64MB, 256MB, 512MB, >> > 1GB, and 2GB. The program got killed in all these cache sizes. In >> > fact, my Linux box became unresponsive when I set GDALSetCacheMax() to >> > 64MB. >> > >> > Thank you. >> > Ozy >> > >> > >> > ------------------------------------------------------------------------ >> > >> > _______________________________________________ >> > gdal-dev mailing list >> > gdal-dev@lists.osgeo.org >> > http://lists.osgeo.org/mailman/listinfo/gdal-dev >> >> >> >
_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev