I took a closer look at the files. It looks like they maxed out the byte size and were truncated. Several NITF images have different rows * columns, but the same byte size.
May not be a GDAL issue. ________________________________ From: gdal-dev <gdal-dev-boun...@lists.osgeo.org> on behalf of ni hao <ni_ha...@hotmail.com> Sent: April 7, 2021 10:52 AM To: Even Rouault <even.roua...@spatialys.com>; gdal-dev@lists.osgeo.org <gdal-dev@lists.osgeo.org> Subject: Re: [gdal-dev] NITF Int16 image size limit Hi Even, It was CSA using their own software. They said that it was fine access the last 2000 lines. But I could not confirm. ________________________________ From: Even Rouault <even.roua...@spatialys.com> Sent: April 7, 2021 10:41 AM To: ni hao <ni_ha...@hotmail.com>; gdal-dev@lists.osgeo.org <gdal-dev@lists.osgeo.org> Subject: Re: [gdal-dev] NITF Int16 image size limit Hum, 4294967295 = 0xffffffff = max uint32 value. This really a suspicious file size that looks like truncation at the 4 GB limit. Are you really really sure that you tested the non-GDAL implementation with that exact file, and not another version that might have been truncated later ? Le 07/04/2021 à 14:59, ni hao a écrit : Hi Even, I am using GDAL 2.4.4. I may try the latest version as you suggested. The NITF file size is 4,294,967,295 bytes The vendor used a non-GDAL implementation, and was able to display all image. Metadata: NITF_ABPP=16 NITF_BLOCKA_BLOCK_COUNT=01 NITF_BLOCKA_BLOCK_INSTANCE_01=01 NITF_BLOCKA_FRFC_LOC_01= NITF_BLOCKA_FRLC_LOC_01= NITF_BLOCKA_LAYOVER_ANGLE_01=000 NITF_BLOCKA_LRFC_LOC_01= NITF_BLOCKA_LRLC_LOC_01= NITF_BLOCKA_L_LINES_01=19694 NITF_BLOCKA_N_GRAY_01=00000 NITF_BLOCKA_SHADOW_ANGLE_01=180 NITF_CCS_COLUMN=0 NITF_CCS_ROW=0 NITF_CLEVEL=07 NITF_ENCRYP=0 NITF_FBKGC= 0, 0, 0 NITF_FDT= NITF_FHDR=NITF02.10 NITF_FSCATP= NITF_FSCAUT= NITF_FSCLAS=U NITF_FSCLSY=CA NITF_FSCLTX= NITF_FSCODE= NITF_FSCOP=00000 NITF_FSCPYS=00000 NITF_FSCRSN= NITF_FSCTLH= NITF_FSCTLN= NITF_FSDCDT= NITF_FSDCTP= NITF_FSDCXM= NITF_FSDG= NITF_FSDGDT= NITF_FSREL= NITF_FSSRDT= NITF_FTITLE= NITF_IALVL=0 NITF_IC=NC NITF_ICAT=SAR NITF_ICORDS=G NITF_IDATIM= NITF_IDLVL= NITF_IGEOLO= NITF_IID1=P1 NITF_IID2= NITF_ILOC_COLUMN=0 NITF_ILOC_ROW=0 NITF_IMAG=1.0 NITF_IMODE=P NITF_IREP=MULTI NITF_ISCATP= NITF_ISCAUT= NITF_ISCLAS=U NITF_ISCLSY=CA NITF_ISCLTX= NITF_ISCODE= NITF_ISCRSN= NITF_ISCTLH= NITF_ISCTLN= NITF_ISDCDT= NITF_ISDCTP= NITF_ISDCXM= NITF_ISDG= NITF_ISDGDT= NITF_ISORCE= NITF_ISREL= NITF_ISSRDT= NITF_ONAME= NITF_OPHONE= NITF_OSTAID=CSA NITF_PJUST=R NITF_PVTYPE=INT NITF_STYPE=BF01 NITF_TGTID= ________________________________ From: Even Rouault <even.roua...@spatialys.com><mailto:even.roua...@spatialys.com> Sent: April 7, 2021 5:23 AM To: ni hao <ni_ha...@hotmail.com><mailto:ni_ha...@hotmail.com>; gdal-dev@lists.osgeo.org<mailto:gdal-dev@lists.osgeo.org> <gdal-dev@lists.osgeo.org><mailto:gdal-dev@lists.osgeo.org> Subject: Re: [gdal-dev] NITF Int16 image size limit Hi, Actually the error message doesn't look like there's a uint32 overflow as it displays a file offset that is beyond the 4 GB limit. Did you check that your file size is at least 4324329780 + 4194302 = 4 328 524 082 bytes ? I guess it is not, or then there's some mysterious error in the low level GDAL I/O layer. So if your file is smaller than that, - either it is corrupted (would be good if you could check that with a non-GDAL implementation) - or there's a bug in the NITF driver in how it computes offsets. Could you report the values of the NITF_ABPP, NITF_IC and NITF_IMODE metadata reported by gdalinfo ? Even Le 07/04/2021 à 02:09, ni hao a écrit : gdalinfo output i: Driver: NITF/National Imagery Transmission Format Files: ##CH_CV_GRD\imagery\12499.ntf Size is 61180, 19694 Coordinate System is `' GCP Projection = GEOGCS["WGS 84", ... ] Band 1 Block=1024x1024 Type=UInt16, ColorInterp=Gray Band 2 Block=1024x1024 Type=UInt16, ColorInterp=Gray Also I am sure it is the UInt16 overflow, as the message is: ERROR 3: Unable to read 4194302 byte block from 4324329780 and 2**32 = 4294967296 ________________________________ From: br...@frogmouth.net<mailto:br...@frogmouth.net> <br...@frogmouth.net><mailto:br...@frogmouth.net> Sent: April 6, 2021 8:17 PM To: Even Rouault <even.roua...@spatialys.com><mailto:even.roua...@spatialys.com>; ni hao <ni_ha...@hotmail.com><mailto:ni_ha...@hotmail.com> Cc: gdal-dev@lists.osgeo.org<mailto:gdal-dev@lists.osgeo.org> <gdal-dev@lists.osgeo.org><mailto:gdal-dev@lists.osgeo.org> Subject: Re: [gdal-dev] NITF Int16 image size limit Also if you can show the headers (e.g. from gdalinfo), that might provide indication of exactly which value is the issue. Brad On 7 Apr. 2021 04:00, Even Rouault <even.roua...@spatialys.com><mailto:even.roua...@spatialys.com> wrote: Please keep the mailing list copied (I re-added it) This is not the most recent one (3.2.2 is), but skimming quickly through release notes, I can't see anything directly related, so it might still be current with the recent versions. Altough I see that a bunch of "avoid unsigned integer overflow" type of fixes have been done since 2.4, so some of them might have accidentally fixed the issue. If you can try with the latest version ( Docker images mentioned at https://github.com/OSGeo/gdal/tree/master/gdal/docker, Conda, etc .), that could be interesting. Le 06/04/2021 à 19:51, ni hao a écrit : Hi Even, The version is 2.4.4. ________________________________ From: Even Rouault <even.roua...@spatialys.com><mailto:even.roua...@spatialys.com> Sent: April 6, 2021 2:46 PM To: ni hao <ni_ha...@hotmail.com><mailto:ni_ha...@hotmail.com>; gdal-dev@lists.osgeo.org<mailto:gdal-dev@lists.osgeo.org> <gdal-dev@lists.osgeo.org><mailto:gdal-dev@lists.osgeo.org> Subject: Re: [gdal-dev] NITF Int16 image size limit Shawn, This sounds more like a unwanted integer overflow somewhere in the NITF driver. Is this with a recent GDAL ? If so, please file a bug at https://github.com/OSGeo/gdal/issues/new with all the details needed. A link to the dataset would be ideal, but otherwise please provide in the ticket description the output of "gdalinfo your.nitf" Even Le 06/04/2021 à 19:35, ni hao a écrit : Hi list, I encountered problem ingesting a large NITF Int16 image with GDAL: NITF 2-band image in Int16 format, 4.3 GB. It has 19690 lines x 61180 pixels. The upper 90% of the image looks fine. But the last 2000 lines fail to load by GDAL. That coincides with the 32-bit boundary. Note that another larger NITF in 32-bit float, 10.1 GB file loads fine. Is there a hard limit on NITF Int16 file size? Thanks, Shawn _______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org<mailto:gdal-dev@lists.osgeo.org> https://lists.osgeo.org/mailman/listinfo/gdal-dev -- http://www.spatialys.com My software is free, but my time generally not. -- http://www.spatialys.com My software is free, but my time generally not. _______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org<mailto:gdal-dev@lists.osgeo.org> https://lists.osgeo.org/mailman/listinfo/gdal-dev -- http://www.spatialys.com My software is free, but my time generally not. -- http://www.spatialys.com My software is free, but my time generally not.
_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
