Re: [gdal-dev] gdalwarp's default nodata value
Joaquim, For the rec, VS2010 and a quite recent trunk version. Ah, this is the reason for the different results we got. On Windows, atof(nan) returns 0... but on Linux it returns a nan number. I've pushed quite a few changes to trunk to improve the situation on Windows and other changes for all platforms to fix the case when we compare pixel values to nodata value and the nodata value is nan. Indeed, the test value == dfNoDataValue that was generally used fails if value == nan and dfNoDataValue == nan, because nan != nan... See http://trac.osgeo.org/gdal/ticket/3576 Best regards, Even ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] gdalwarp's default nodata value
On 13-05-2010 18:38, Even Rouault wrote: Joaquim, For the rec, VS2010 and a quite recent trunk version. Ah, this is the reason for the different results we got. On Windows, atof(nan) returns 0... but on Linux it returns a nan number. I've pushed quite a few changes to trunk to improve the situation on Windows and other changes for all platforms to fix the case when we compare pixel values to nodata value and the nodata value is nan. Indeed, the test value == dfNoDataValue that was generally used fails if value == nan and dfNoDataValue == nan, because nan != nan... Oui, c'est le danger des nains Thanks for the info (and work of course) Joaquim ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] gdalwarp's default nodata value
On 12-05-2010 05:55, Chaitanya kumar CH wrote: Joaquim, On Wed, May 12, 2010 at 5:07 AM, Joaquim Luis jl...@ualg.pt mailto:jl...@ualg.pt wrote: Hi, Before filling a ticket I would like to ask here if this gdalwarp behavior is the intended one. When I convert a grid from geogs to UTM the nodatavalues are filled with zeros. I get the expected behaviour if I use the -dstnodata with a numeric value, but I found no way tom tell it use NaN. Summary This puts zeros on the nodata zone, but I don't find it correct as 0 is not exactly a natural nodata value. For my habits NaN is the natural no data value. NaN should always be treated as a special case in coding. Imagine performing a type conversion. Since we usually deal with real world data, we know the data value range. We should be able to choose a nodata value not in the data range. Hi Chaitanya, Taking your argument of the real world data, it is why the default choice of zero for nodata is one of worst possible choices. At least for the case of floating point data. Imagine that the input grid has zeros as perfectly valid values, how will any application be able to distinguish between the good and the bad zeros on the warped result? gdalwarp -s_srs +proj=latlong -t_srs +proj=utm +zone=29 +datum=WGS84 swath_grid.grd lixo_utm.tiff Furthermore when I load the lixo_utm.tiff in Mirone is does not recognize a nodata value, whilst if I do this instead gdalwarp -s_srs +proj=latlong -t_srs +proj=utm +zone=29 +datum=WGS84 -dstnodata 1 swath_grid.grd lixo_utm.tiff than 1 is recognized as the nodata. I have not investigated the metadata to see why the 0 is not set to represent the nodata. Perhaps swath_grid.grd doesn't have a nodata value set. The grid was created by GMT (it always sets a nodata value defaulting no NaN) but that is not the problem. I dug a bit more on this and actually there is no problem at all in what respects recognizing the nodata value even when I let gdalwarp use the default value of zero. But my real problem is with my gdalwarp_mex MEX file used in Mirone. Even if I add this for (i = 0; i nBands; i++) GDALSetRasterNoDataValue( GDALGetRasterBand(hDstDS, i+1),999.0); the warped dataset has the correct nodata value in its metadata but the array still has zeros where it should have 999.0 I checked again against the gdalwarp.cpp code and the only difference I'm able to identify is that in gdalwarp_mex I'm using the MEM driver (I have to since data never lands on hard disk). I'm lost on this one. Joaquim ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] gdalwarp's default nodata value
Joaquim, GDALSetRasterNoDataValue() sets only the metadata. It doesn't actually change the nodata pixels. We need to process each pixel to do this. Or you could just create a VRT mentioning the source and vrt nodata values. http://www.gdal.org/gdal_vrttut.html http://www.gdal.org/gdalbuildvrt.html On Wed, May 12, 2010 at 6:33 PM, Joaquim Luis jl...@ualg.pt wrote: On 12-05-2010 05:55, Chaitanya kumar CH wrote: Joaquim, On Wed, May 12, 2010 at 5:07 AM, Joaquim Luis jl...@ualg.pt wrote: Hi, Before filling a ticket I would like to ask here if this gdalwarp behavior is the intended one. When I convert a grid from geogs to UTM the nodatavalues are filled with zeros. I get the expected behaviour if I use the -dstnodata with a numeric value, but I found no way tom tell it use NaN. Summary This puts zeros on the nodata zone, but I don't find it correct as 0 is not exactly a natural nodata value. For my habits NaN is the natural no data value. NaN should always be treated as a special case in coding. Imagine performing a type conversion. Since we usually deal with real world data, we know the data value range. We should be able to choose a nodata value not in the data range. Hi Chaitanya, Taking your argument of the real world data, it is why the default choice of zero for nodata is one of worst possible choices. At least for the case of floating point data. Imagine that the input grid has zeros as perfectly valid values, how will any application be able to distinguish between the good and the bad zeros on the warped result? gdalwarp -s_srs +proj=latlong -t_srs +proj=utm +zone=29 +datum=WGS84 swath_grid.grd lixo_utm.tiff Furthermore when I load the lixo_utm.tiff in Mirone is does not recognize a nodata value, whilst if I do this instead gdalwarp -s_srs +proj=latlong -t_srs +proj=utm +zone=29 +datum=WGS84 -dstnodata 1 swath_grid.grd lixo_utm.tiff than 1 is recognized as the nodata. I have not investigated the metadata to see why the 0 is not set to represent the nodata. Perhaps swath_grid.grd doesn't have a nodata value set. The grid was created by GMT (it always sets a nodata value defaulting no NaN) but that is not the problem. I dug a bit more on this and actually there is no problem at all in what respects recognizing the nodata value even when I let gdalwarp use the default value of zero. But my real problem is with my gdalwarp_mex MEX file used in Mirone. Even if I add this for (i = 0; i nBands; i++) GDALSetRasterNoDataValue( GDALGetRasterBand(hDstDS, i+1),999.0); the warped dataset has the correct nodata value in its metadata but the array still has zeros where it should have 999.0 I checked again against the gdalwarp.cpp code and the only difference I'm able to identify is that in gdalwarp_mex I'm using the MEM driver (I have to since data never lands on hard disk). I'm lost on this one. Joaquim -- Best regards, Chaitanya kumar CH. /tʃaɪθənjə/ /kʊmɑr/ +91-9848167848 17.2416N 80.1426E ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] gdalwarp's default nodata value
On 12-05-2010 15:58, Chaitanya kumar CH wrote: Joaquim, GDALSetRasterNoDataValue() sets only the metadata. It doesn't actually change the nodata pixels. We need to process each pixel to do this. Or you could just create a VRT mentioning the source and vrt nodata values. Chaitanya, Thanks, but sorry I don't get it. I thought that it was up to the warping machinery to use the nodata value that it was instructed to and put them in the array positions of nodatavalue. If, at a certain point, it decides to put zeros at some memory addresses than at that point it should simply put the nodata value. It seams quite logic this reasoning no? Please no VRTs since this is a pure memory operation. http://www.gdal.org/gdal_vrttut.html http://www.gdal.org/gdalbuildvrt.html On Wed, May 12, 2010 at 6:33 PM, Joaquim Luis jl...@ualg.pt mailto:jl...@ualg.pt wrote: On 12-05-2010 05:55, Chaitanya kumar CH wrote: Joaquim, On Wed, May 12, 2010 at 5:07 AM, Joaquim Luis jl...@ualg.pt mailto:jl...@ualg.pt wrote: Hi, Before filling a ticket I would like to ask here if this gdalwarp behavior is the intended one. When I convert a grid from geogs to UTM the nodatavalues are filled with zeros. I get the expected behaviour if I use the -dstnodata with a numeric value, but I found no way tom tell it use NaN. Summary This puts zeros on the nodata zone, but I don't find it correct as 0 is not exactly a natural nodata value. For my habits NaN is the natural no data value. NaN should always be treated as a special case in coding. Imagine performing a type conversion. Since we usually deal with real world data, we know the data value range. We should be able to choose a nodata value not in the data range. Hi Chaitanya, Taking your argument of the real world data, it is why the default choice of zero for nodata is one of worst possible choices. At least for the case of floating point data. Imagine that the input grid has zeros as perfectly valid values, how will any application be able to distinguish between the good and the bad zeros on the warped result? gdalwarp -s_srs +proj=latlong -t_srs +proj=utm +zone=29 +datum=WGS84 swath_grid.grd lixo_utm.tiff Furthermore when I load the lixo_utm.tiff in Mirone is does not recognize a nodata value, whilst if I do this instead gdalwarp -s_srs +proj=latlong -t_srs +proj=utm +zone=29 +datum=WGS84 -dstnodata 1 swath_grid.grd lixo_utm.tiff than 1 is recognized as the nodata. I have not investigated the metadata to see why the 0 is not set to represent the nodata. Perhaps swath_grid.grd doesn't have a nodata value set. The grid was created by GMT (it always sets a nodata value defaulting no NaN) but that is not the problem. I dug a bit more on this and actually there is no problem at all in what respects recognizing the nodata value even when I let gdalwarp use the default value of zero. But my real problem is with my gdalwarp_mex MEX file used in Mirone. Even if I add this for (i = 0; i nBands; i++) GDALSetRasterNoDataValue( GDALGetRasterBand(hDstDS, i+1),999.0); the warped dataset has the correct nodata value in its metadata but the array still has zeros where it should have 999.0 I checked again against the gdalwarp.cpp code and the only difference I'm able to identify is that in gdalwarp_mex I'm using the MEM driver (I have to since data never lands on hard disk). I'm lost on this one. Joaquim -- Best regards, Chaitanya kumar CH. /tʃaɪθənjə/ /kʊmɑr/ +91-9848167848 17.2416N 80.1426E ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] gdalwarp's default nodata value
Joaquim, I believe what you see is the expected behaviour. You must keep in mind that there are 2 notions : 1) nodata value -- setting it is no more than just setting a tag on the raster band. It doesn't change the pixel values by itself as Chaintanya explained 2) initialization of the pixel data of the target image with a given value When you use the -dstnodata option, gdalwarp will automagically do both : 1) Set the target nodata value 2) Initialize the target image with that nodata value (remind of removing the target file if it already exists) If you specify nothing, then 0 is used to fill the target image but the nodata tag is not set. The initialization of the pixel data of the target image is in fact done through the INIT_DEST warping option. See http://gdal.org/structGDALWarpOptions.html#0ed77f9917bb96c7a9aabd73d4d06e08 For example, gdalwarp src.tif dst.tif -wo INIT_DEST=255 If you specify no INIT_DEST warping option, then gdalwarp uses the dstnodata value if it's specified with -dstnodata and set it as the INIT_DEST value too. I've also tested -dstnodata nan and to my great surprise, it actually works ! However I'm not sure if all applications can deal with the nan string put in the 42113 tag used by GDAL for the Nodata value in TIFF. So maybe I've not understood what your problem is, but everything seems to work as expected on my side. Best regards, Even Le Wednesday 12 May 2010 01:37:07 Joaquim Luis, vous avez écrit : Hi, Before filling a ticket I would like to ask here if this gdalwarp behavior is the intended one. When I convert a grid from geogs to UTM the nodatavalues are filled with zeros. I get the expected behaviour if I use the -dstnodata with a numeric value, but I found no way tom tell it use NaN. Summary This puts zeros on the nodata zone, but I don't find it correct as 0 is not exactly a natural nodata value. For my habits NaN is the natural no data value. gdalwarp -s_srs +proj=latlong -t_srs +proj=utm +zone=29 +datum=WGS84 swath_grid.grd lixo_utm.tiff Furthermore when I load the lixo_utm.tiff in Mirone is does not recognize a nodata value, whilst if I do this instead gdalwarp -s_srs +proj=latlong -t_srs +proj=utm +zone=29 +datum=WGS84 -dstnodata 1 swath_grid.grd lixo_utm.tiff than 1 is recognized as the nodata. I have not investigated the metadata to see why the 0 is not set to represent the nodata. Thanks Joaquim Luis ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] gdalwarp's default nodata value
Even,
Thanks, I understand it better now and made some advances but I'm still facing
some troubles.
For example, if I do this
papszWarpOptions = CSLSetNameValue(papszWarpOptions, INIT_DEST, 300 );
psWO-papszWarpOptions = CSLDuplicate(papszWarpOptions);
than it works fine.
However, if I do (remember that I want NaNs)
papszWarpOptions = CSLSetNameValue(papszWarpOptions, INIT_DEST, nan );
than I get zeros again in place of the nodata.
The link you pointed me says
...or INIT_DEST=NO_DATA: This option forces the destination image to be initialized to the
indicated value (for all bands) or indicates that it should be initialized to the NO_DATA value in
padfDstNoDataReal/padfDstNoDataImag.
So I next tried
papszWarpOptions = CSLSetNameValue(papszWarpOptions, INIT_DEST, NO_DATA );
...
psWO-padfSrcNoDataReal = (double *)
CPLMalloc(psWO-nBandCount*sizeof(double));
psWO-padfSrcNoDataImag = (double *)
CPLMalloc(psWO-nBandCount*sizeof(double));
for (i = 0; i nBands; i++) {
psWO-padfSrcNoDataReal[i] = 500.;
psWO-padfSrcNoDataImag[i] = 0.0;
GDALSetRasterNoDataValue( GDALGetRasterBand(hDstDS, i+1),
pdfDstNodata[0]);
}
and guess what? bloody zeros again. So I'm not able to get the NaNs (tried with 500 to see if this
solution works)
I tried to see what gdalwarp does ... and another problem. What is this?
gdalwarp -of netCDF -s_srs +proj=latlong -t_srs +proj=utm +zone=29 +datum=WGS84 swath_grid.grd
lixo_utm.grd
Output driver `netCDF' not recognised or does not support
direct output file creation. The following format drivers are configured
and support direct output:
VRT: Virtual Raster
GTiff: GeoTIFF
NITF: National Imagery Transmission Format
HFA: Erdas Imagine Images (.img)
ELAS: ELAS
MEM: In Memory Raster
BMP: MS Windows Device Independent Bitmap
PCIDSK: PCIDSK Database File
ILWIS: ILWIS Raster Map
SGI: SGI Image File Format 1.0
Leveller: Leveller heightfield
Terragen: Terragen heightfield
HDF4Image: HDF4 Dataset
ERS: ERMapper .ers Labelled
JP2ECW: ERMapper JPEG2000
RMF: Raster Matrix Format
RST: Idrisi Raster A.1
INGR: Intergraph Raster
GSBG: Golden Software Binary Grid (.grd)
PNM: Portable Pixmap Format (netpbm)
ENVI: ENVI .hdr Labelled
EHdr: ESRI .hdr Labelled
PAux: PCI .aux Labelled
MFF: Vexcel MFF Raster
MFF2: Vexcel MFF2 (HKV) Raster
BT: VTP .bt (Binary Terrain) 1.3 Format
IDA: Image Data and Analysis
ADRG: ARC Digitized Raster Graphics
SAGA: SAGA GIS Binary Grid (.sdat)
I surely have the netCDF driver. The swath_grid.grd file is in netCDF and the above command works
for the default output type (GeoTiff).
Furthermore, if I ask for the drivers I have (I'm using FWTools here)
gdalwarp --formats
Supported Formats:
VRT (rw+): Virtual Raster
GTiff (rw+v): GeoTIFF
NITF (rw+v): National Imagery Transmission Format
RPFTOC (ro): Raster Product Format TOC format
HFA (rw+v): Erdas Imagine Images (.img)
SAR_CEOS (ro): CEOS SAR Image
CEOS (ro): CEOS Image
JAXAPALSAR (ro): JAXA PALSAR Product Reader (Level 1.1/1.5)
GFF (rov): Ground-based SAR Applications Testbed File Format (.gff)
ELAS (rw+): ELAS
AIG (ro): Arc/Info Binary Grid
AAIGrid (rwv): Arc/Info ASCII Grid
SDTS (ro): SDTS Raster
OGDI (ro): OGDI Bridge
DTED (rwv): DTED Elevation Raster
PNG (rwv): Portable Network Graphics
JPEG (rwv): JPEG JFIF
MEM (rw+): In Memory Raster
JDEM (ro): Japanese DEM (.mem)
GIF (rwv): Graphics Interchange Format (.gif)
BIGGIF (rov): Graphics Interchange Format (.gif)
ESAT (ro): Envisat Image Format
BSB (ro): Maptech BSB Nautical Charts
XPM (rw): X11 PixMap Format
BMP (rw+v): MS Windows Device Independent Bitmap
DIMAP (ro): SPOT DIMAP
AirSAR (ro): AirSAR Polarimetric Image
RS2 (ro): RadarSat 2 XML Product
PCIDSK (rw+v): PCIDSK Database File
PCRaster (rw): PCRaster Raster File
ILWIS (rw+v): ILWIS Raster Map
SGI (rw+): SGI Image File Format 1.0
SRTMHGT (rwv): SRTMHGT File Format
Leveller (rw+): Leveller heightfield
Terragen (rw+): Terragen heightfield
GMT (rw): GMT NetCDF Grid Format
netCDF (rw): Network Common Data Format
HDF4 (ro): Hierarchical Data Format Release 4
HDF4Image (rw+): HDF4 Dataset
ISIS3 (ro): USGS Astrogeology ISIS cube (Version 3)
ISIS2 (ro): USGS Astrogeology ISIS cube (Version 2)
PDS (ro): NASA Planetary Data System
TIL (ro): EarthWatch .TIL
ERS (rw+): ERMapper .ers Labelled
ECW (rw): ERMapper Compressed Wavelets
JP2ECW (rw+): ERMapper JPEG2000
L1B (ro): NOAA Polar Orbiter Level 1b Data Set
FIT (rw): FIT Image
GRIB (ro): GRIdded Binary (.grb)
MrSID (ro): Multi-resolution Seamless Image Database (MrSID)
JP2MrSID (ro): MrSID JPEG2000
RMF (rw+): Raster Matrix Format
WCS (ro): OGC Web Coverage Service
WMS (ro): OGC Web Map Service
MSGN (ro): EUMETSAT Archive native (.nat)
RST (rw+): Idrisi Raster A.1
INGR (rw+): Intergraph Raster
GSAG (rw): Golden
Re: [gdal-dev] gdalwarp's default nodata value
Le Wednesday 12 May 2010 23:15:08 Joaquim Luis, vous avez écrit :
Even,
Thanks, I understand it better now and made some advances but I'm still
facing some troubles.
For example, if I do this
papszWarpOptions = CSLSetNameValue(papszWarpOptions, INIT_DEST, 300 );
psWO-papszWarpOptions = CSLDuplicate(papszWarpOptions);
than it works fine.
However, if I do (remember that I want NaNs)
papszWarpOptions = CSLSetNameValue(papszWarpOptions, INIT_DEST, nan );
than I get zeros again in place of the nodata.
Works for me. You'll have to use your debugger I'm afraid
The link you pointed me says
...or INIT_DEST=NO_DATA: This option forces the destination image to be
initialized to the indicated value (for all bands) or indicates that it
should be initialized to the NO_DATA value in
padfDstNoDataReal/padfDstNoDataImag.
So I next tried
papszWarpOptions = CSLSetNameValue(papszWarpOptions, INIT_DEST, NO_DATA
); ...
psWO-padfSrcNoDataReal = (double *)
CPLMalloc(psWO-nBandCount*sizeof(double)); psWO-padfSrcNoDataImag =
(double *) CPLMalloc(psWO-nBandCount*sizeof(double)); for (i = 0; i
nBands; i++) {
psWO-padfSrcNoDataReal[i] = 500.;
psWO-padfSrcNoDataImag[i] = 0.0;
GDALSetRasterNoDataValue( GDALGetRasterBand(hDstDS, i+1),
pdfDstNodata[0]); }
Hum, do you realize that you set padfSrcNoDataReal ? INIT_DEST=NO_DATA will
fetch from padfDstNoDataReal as stated in the above doc...
I tried to see what gdalwarp does ... and another problem. What is this?
gdalwarp -of netCDF -s_srs +proj=latlong -t_srs +proj=utm +zone=29
+datum=WGS84 swath_grid.grd lixo_utm.grd
Output driver `netCDF' not recognised or does not support
direct output file creation. The following format drivers are configured
and support direct output:
Read the error message again ;-) does not support direct output file
creation-- netCDF driver indeed only supports CreateCopy(), not Create()
(-- random access) which is required by gdalwarp.
The drivers that are supported as output driver for gdalwarp will have a '+'
in the output of gdalinfo --formats. Look at the difference betwenn GeoTIFF
and netCDF:
GTiff (rw+v): GeoTIFF
netCDF (rw): Network Common Data Format
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] gdalwarp's default nodata value
papszWarpOptions = CSLSetNameValue(papszWarpOptions, INIT_DEST, nan );
than I get zeros again in place of the nodata.
Works for me. You'll have to use your debugger I'm afraid
For the rec, VS2010 and a quite recent trunk version.
papszWarpOptions = CSLSetNameValue(papszWarpOptions, INIT_DEST, NO_DATA
); ...
psWO-padfSrcNoDataReal = (double *)
CPLMalloc(psWO-nBandCount*sizeof(double)); psWO-padfSrcNoDataImag =
(double *) CPLMalloc(psWO-nBandCount*sizeof(double)); for (i = 0; i
nBands; i++) {
psWO-padfSrcNoDataReal[i] = 500.;
psWO-padfSrcNoDataImag[i] = 0.0;
GDALSetRasterNoDataValue( GDALGetRasterBand(hDstDS, i+1),
pdfDstNodata[0]); }
Hum, do you realize that you set padfSrcNoDataReal ? INIT_DEST=NO_DATA will
fetch from padfDstNoDataReal as stated in the above doc...
Ghrr, blind copy-paste is-what-it-was
THANKS
psWO-padfDstNoDataReal[i] = mxGetNaN();// ( mxGetNaN() -- a
Matlab API function)
Works fine now.
Joaquim
Read the error message again ;-) does not support direct output file
creation-- netCDF driver indeed only supports CreateCopy(), not Create()
(-- random access) which is required by gdalwarp.
The drivers that are supported as output driver for gdalwarp will have a '+'
in the output of gdalinfo --formats. Look at the difference betwenn GeoTIFF
and netCDF:
GTiff (rw+v): GeoTIFF
netCDF (rw): Network Common Data Format
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
