[gdal-dev] Fwd: [GRASS-user] Please check your trac settings for email notifications
Hi, Below information is also true for GDAL Trac. Even -- Message transmis -- Sujet : [GRASS-user] Please check your trac settings for email notifications Date : lundi 15 juin 2015, 11:14:29 De : Markus Neteler nete...@osgeo.org À : GRASS developers list grass-...@lists.osgeo.org, GRASS user list grass- u...@lists.osgeo.org Hi all, FYI recently, the OSGeo trac server got updated. Apparently the personal email preferences didn't survive this update. If you want to again receive email notification upon trac ticket updates, you need to login and re-enter your name and email address here: https://trac.osgeo.org/grass/prefs Preferences -- Full name: -- Email address: This information is used to associate your login name with your email address and full name, which is used for email notification and RSS feeds, for example. Best, Markus PS: thanks to Moritz Lennert for identifying this issue. ___ grass-user mailing list grass-u...@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user --- -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] GDALWarp on a specific overview of a dataset
Hi,
I am having a bit of trouble with warping a dataset (an ECW file) with
multiple overviews. The base file is huge, so I am selecting the overview
that matches closest to my output resolution and makes an in-memory VRT of
this, consisting of the overview bands of the original image.
This seems to only work for some overview levels, and not for others. I get
results that appear to be made from the correct sets of colors basically,
but way out of scale and postion. Some times it seems to work nicely
though, so I suspect that there is some memory overwriting issue or
something here.
This is the part of the code that builds the overview VRT. The input
dataset is _ds, an
// Create a VRT of the closest matching overlay
GDALDriver *vrtDriver = (GDALDriver *) GDALGetDriverByName( VRT );
vrtDS = vrtDriver-Create(, ovrW, ovrH, _numBands, _dataType, NULL);
GDALSetProjection(vrtDS, _srcProjectionWKT.c_str());
ouble geoTransform[6]; _ds-GetGeoTransform(geoTransform); double
basePixelSizeX = geoTransform[XFM_PIXEL_SIZE_EW]; double basePixelSizeY =
geoTransform[XFM_PIXEL_SIZE_NS];
geoTransform[XFM_PIXEL_SIZE_EW] = scaleX*basePixelSizeX;
geoTransform[XFM_PIXEL_SIZE_NS] = scaleY*basePixelSizeY;
vrtDS-SetGeoTransform(geoTransform);
for(int i = 1; i = _numBands; i++)
{
GDALRasterBand* srcRootBand = _ds-GetRasterBand(i);
GDALRasterBand* srcBand = srcRootBand-GetOverview(overviewId);
VRTSourcedRasterBand* vrtBand =
(VRTSourcedRasterBand*)vrtDS-GetRasterBand(i);
vrtBand-AddSimpleSource(srcBand);
}
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GDALWarp on a specific overview of a dataset
Thomas,
2 suggestions to help narrowing down the issue :
- try gdalwarp of GDAL 2.0 that can select the most appropriate overview
given the target resolution / raster size
- try with the ECW translated as a GeoTIFF to check if it isn't an issue with
the ECW driver itself.
Even
Hi,
I am having a bit of trouble with warping a dataset (an ECW file) with
multiple overviews. The base file is huge, so I am selecting the overview
that matches closest to my output resolution and makes an in-memory VRT of
this, consisting of the overview bands of the original image.
This seems to only work for some overview levels, and not for others. I get
results that appear to be made from the correct sets of colors basically,
but way out of scale and postion. Some times it seems to work nicely
though, so I suspect that there is some memory overwriting issue or
something here.
This is the part of the code that builds the overview VRT. The input
dataset is _ds, an
// Create a VRT of the closest matching overlay
GDALDriver *vrtDriver = (GDALDriver *) GDALGetDriverByName( VRT );
vrtDS = vrtDriver-Create(, ovrW, ovrH, _numBands, _dataType, NULL);
GDALSetProjection(vrtDS, _srcProjectionWKT.c_str());
ouble geoTransform[6]; _ds-GetGeoTransform(geoTransform); double
basePixelSizeX = geoTransform[XFM_PIXEL_SIZE_EW]; double basePixelSizeY =
geoTransform[XFM_PIXEL_SIZE_NS];
geoTransform[XFM_PIXEL_SIZE_EW] = scaleX*basePixelSizeX;
geoTransform[XFM_PIXEL_SIZE_NS] = scaleY*basePixelSizeY;
vrtDS-SetGeoTransform(geoTransform);
for(int i = 1; i = _numBands; i++)
{
GDALRasterBand* srcRootBand = _ds-GetRasterBand(i);
GDALRasterBand* srcBand = srcRootBand-GetOverview(overviewId);
VRTSourcedRasterBand* vrtBand =
(VRTSourcedRasterBand*)vrtDS-GetRasterBand(i);
vrtBand-AddSimpleSource(srcBand);
}
--
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GDALWarp on a specific overview of a dataset
Argh, I managed to send it while editing :-)
Anway, input to this code section is the wanted overview level and the
derived scale factors in X- and Y-direction. Output is the vrtDS dataset
that is used in the GDAL Warp api later.
Are there any obvious mistakes I've made here?
I see that in GDAL 2.0 there will be a new class for handling this, the
GDALOverviewDaset. It's a bit early for me to move to 2.0 yet, but maybe I
should just make a similar dataset type in my own project. Would that be
more efficient than going via a VRT?
- Thomas
On Mon, Jun 15, 2015 at 12:07 PM, Thomas Sevaldrud tho...@silentwings.no
wrote:
Hi,
I am having a bit of trouble with warping a dataset (an ECW file) with
multiple overviews. The base file is huge, so I am selecting the overview
that matches closest to my output resolution and makes an in-memory VRT of
this, consisting of the overview bands of the original image.
This seems to only work for some overview levels, and not for others. I
get results that appear to be made from the correct sets of colors
basically, but way out of scale and postion. Some times it seems to work
nicely though, so I suspect that there is some memory overwriting issue or
something here.
This is the part of the code that builds the overview VRT. The input
dataset is _ds, an
// Create a VRT of the closest matching overlay
GDALDriver *vrtDriver = (GDALDriver *) GDALGetDriverByName( VRT );
vrtDS = vrtDriver-Create(, ovrW, ovrH, _numBands, _dataType, NULL);
GDALSetProjection(vrtDS, _srcProjectionWKT.c_str());
ouble geoTransform[6]; _ds-GetGeoTransform(geoTransform); double
basePixelSizeX = geoTransform[XFM_PIXEL_SIZE_EW]; double basePixelSizeY =
geoTransform[XFM_PIXEL_SIZE_NS];
geoTransform[XFM_PIXEL_SIZE_EW] = scaleX*basePixelSizeX;
geoTransform[XFM_PIXEL_SIZE_NS] = scaleY*basePixelSizeY;
vrtDS-SetGeoTransform(geoTransform);
for(int i = 1; i = _numBands; i++)
{
GDALRasterBand* srcRootBand = _ds-GetRasterBand(i);
GDALRasterBand* srcBand = srcRootBand-GetOverview(overviewId);
VRTSourcedRasterBand* vrtBand =
(VRTSourcedRasterBand*)vrtDS-GetRasterBand(i);
vrtBand-AddSimpleSource(srcBand);
}
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GDALWarp on a specific overview of a dataset
When using the same image in GeoTIFF with overviews, it works correctly,
which may indicate something with the ECW drivers. I suppose it must be
something triggered by something weird I'm doing though, or else I would
think that other people would have the same problem.
The weird thing is that if I simply add an extra band to the VRT it works
with ECW as well... This extra band is never used for anything and has no
source, but with it, I get the correct image.
I can't switch to GDAL 2.0 yet, but I'll try to isolate the code from the
GDALOverviewDataset class and use that instead of the VRT approach, and see
if that makes any difference.
- Thomas
On Mon, Jun 15, 2015 at 12:19 PM, Even Rouault even.roua...@spatialys.com
wrote:
Thomas,
2 suggestions to help narrowing down the issue :
- try gdalwarp of GDAL 2.0 that can select the most appropriate overview
given the target resolution / raster size
- try with the ECW translated as a GeoTIFF to check if it isn't an issue
with
the ECW driver itself.
Even
Hi,
I am having a bit of trouble with warping a dataset (an ECW file) with
multiple overviews. The base file is huge, so I am selecting the overview
that matches closest to my output resolution and makes an in-memory VRT
of
this, consisting of the overview bands of the original image.
This seems to only work for some overview levels, and not for others. I
get
results that appear to be made from the correct sets of colors basically,
but way out of scale and postion. Some times it seems to work nicely
though, so I suspect that there is some memory overwriting issue or
something here.
This is the part of the code that builds the overview VRT. The input
dataset is _ds, an
// Create a VRT of the closest matching overlay
GDALDriver *vrtDriver = (GDALDriver *) GDALGetDriverByName( VRT );
vrtDS = vrtDriver-Create(, ovrW, ovrH, _numBands, _dataType, NULL);
GDALSetProjection(vrtDS, _srcProjectionWKT.c_str());
ouble geoTransform[6]; _ds-GetGeoTransform(geoTransform); double
basePixelSizeX = geoTransform[XFM_PIXEL_SIZE_EW]; double basePixelSizeY =
geoTransform[XFM_PIXEL_SIZE_NS];
geoTransform[XFM_PIXEL_SIZE_EW] = scaleX*basePixelSizeX;
geoTransform[XFM_PIXEL_SIZE_NS] = scaleY*basePixelSizeY;
vrtDS-SetGeoTransform(geoTransform);
for(int i = 1; i = _numBands; i++)
{
GDALRasterBand* srcRootBand = _ds-GetRasterBand(i);
GDALRasterBand* srcBand = srcRootBand-GetOverview(overviewId);
VRTSourcedRasterBand* vrtBand =
(VRTSourcedRasterBand*)vrtDS-GetRasterBand(i);
vrtBand-AddSimpleSource(srcBand);
}
--
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GDALWarp on a specific overview of a dataset
Le lundi 15 juin 2015 15:51:37, Thomas Sevaldrud a écrit :
When using the same image in GeoTIFF with overviews, it works correctly,
which may indicate something with the ECW drivers. I suppose it must be
something triggered by something weird I'm doing though, or else I would
think that other people would have the same problem.
It is sometimes surprising that some bugs remain unreported for long!
The weird thing is that if I simply add an extra band to the VRT it works
with ECW as well... This extra band is never used for anything and has no
source, but with it, I get the correct image.
Interesting. My guess is that you must hit
https://trac.osgeo.org/gdal/ticket/5954
I can't switch to GDAL 2.0 yet, but I'll try to isolate the code from the
GDALOverviewDataset class and use that instead of the VRT approach, and see
if that makes any difference.
Latest code from 1.11 branch has the fix for the above ticket.
- Thomas
On Mon, Jun 15, 2015 at 12:19 PM, Even Rouault even.roua...@spatialys.com
wrote:
Thomas,
2 suggestions to help narrowing down the issue :
- try gdalwarp of GDAL 2.0 that can select the most appropriate overview
given the target resolution / raster size
- try with the ECW translated as a GeoTIFF to check if it isn't an issue
with
the ECW driver itself.
Even
Hi,
I am having a bit of trouble with warping a dataset (an ECW file) with
multiple overviews. The base file is huge, so I am selecting the
overview that matches closest to my output resolution and makes an
in-memory VRT
of
this, consisting of the overview bands of the original image.
This seems to only work for some overview levels, and not for others. I
get
results that appear to be made from the correct sets of colors
basically, but way out of scale and postion. Some times it seems to
work nicely though, so I suspect that there is some memory overwriting
issue or something here.
This is the part of the code that builds the overview VRT. The input
dataset is _ds, an
// Create a VRT of the closest matching overlay
GDALDriver *vrtDriver = (GDALDriver *) GDALGetDriverByName( VRT );
vrtDS = vrtDriver-Create(, ovrW, ovrH, _numBands, _dataType, NULL);
GDALSetProjection(vrtDS, _srcProjectionWKT.c_str());
ouble geoTransform[6]; _ds-GetGeoTransform(geoTransform); double
basePixelSizeX = geoTransform[XFM_PIXEL_SIZE_EW]; double basePixelSizeY
= geoTransform[XFM_PIXEL_SIZE_NS];
geoTransform[XFM_PIXEL_SIZE_EW] = scaleX*basePixelSizeX;
geoTransform[XFM_PIXEL_SIZE_NS] = scaleY*basePixelSizeY;
vrtDS-SetGeoTransform(geoTransform);
for(int i = 1; i = _numBands; i++)
{
GDALRasterBand* srcRootBand = _ds-GetRasterBand(i);
GDALRasterBand* srcBand = srcRootBand-GetOverview(overviewId);
VRTSourcedRasterBand* vrtBand =
(VRTSourcedRasterBand*)vrtDS-GetRasterBand(i);
vrtBand-AddSimpleSource(srcBand);
}
--
Spatialys - Geospatial professional services
http://www.spatialys.com
--
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] GDALWarp on a specific overview of a dataset
Ah, great! That fixed it :-)
Is there an official 1.11.3 release in the works? It's a bit of hassle for
me to integrate it into our build system here, so if there's a new official
version out soon I probably won't bother to integrate the current snapshot.
Thanks so much for your help, Even!
- Thomas
On Mon, Jun 15, 2015 at 4:06 PM, Even Rouault even.roua...@spatialys.com
wrote:
Le lundi 15 juin 2015 15:51:37, Thomas Sevaldrud a écrit :
When using the same image in GeoTIFF with overviews, it works correctly,
which may indicate something with the ECW drivers. I suppose it must be
something triggered by something weird I'm doing though, or else I would
think that other people would have the same problem.
It is sometimes surprising that some bugs remain unreported for long!
The weird thing is that if I simply add an extra band to the VRT it works
with ECW as well... This extra band is never used for anything and has no
source, but with it, I get the correct image.
Interesting. My guess is that you must hit
https://trac.osgeo.org/gdal/ticket/5954
I can't switch to GDAL 2.0 yet, but I'll try to isolate the code from the
GDALOverviewDataset class and use that instead of the VRT approach, and
see
if that makes any difference.
Latest code from 1.11 branch has the fix for the above ticket.
- Thomas
On Mon, Jun 15, 2015 at 12:19 PM, Even Rouault
even.roua...@spatialys.com
wrote:
Thomas,
2 suggestions to help narrowing down the issue :
- try gdalwarp of GDAL 2.0 that can select the most appropriate
overview
given the target resolution / raster size
- try with the ECW translated as a GeoTIFF to check if it isn't an
issue
with
the ECW driver itself.
Even
Hi,
I am having a bit of trouble with warping a dataset (an ECW file)
with
multiple overviews. The base file is huge, so I am selecting the
overview that matches closest to my output resolution and makes an
in-memory VRT
of
this, consisting of the overview bands of the original image.
This seems to only work for some overview levels, and not for
others. I
get
results that appear to be made from the correct sets of colors
basically, but way out of scale and postion. Some times it seems to
work nicely though, so I suspect that there is some memory
overwriting
issue or something here.
This is the part of the code that builds the overview VRT. The input
dataset is _ds, an
// Create a VRT of the closest matching overlay
GDALDriver *vrtDriver = (GDALDriver *) GDALGetDriverByName( VRT );
vrtDS = vrtDriver-Create(, ovrW, ovrH, _numBands, _dataType,
NULL);
GDALSetProjection(vrtDS, _srcProjectionWKT.c_str());
ouble geoTransform[6]; _ds-GetGeoTransform(geoTransform); double
basePixelSizeX = geoTransform[XFM_PIXEL_SIZE_EW]; double
basePixelSizeY
= geoTransform[XFM_PIXEL_SIZE_NS];
geoTransform[XFM_PIXEL_SIZE_EW] = scaleX*basePixelSizeX;
geoTransform[XFM_PIXEL_SIZE_NS] = scaleY*basePixelSizeY;
vrtDS-SetGeoTransform(geoTransform);
for(int i = 1; i = _numBands; i++)
{
GDALRasterBand* srcRootBand = _ds-GetRasterBand(i);
GDALRasterBand* srcBand = srcRootBand-GetOverview(overviewId);
VRTSourcedRasterBand* vrtBand =
(VRTSourcedRasterBand*)vrtDS-GetRasterBand(i);
vrtBand-AddSimpleSource(srcBand);
}
--
Spatialys - Geospatial professional services
http://www.spatialys.com
--
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Call for discussion on RFC 26: GDAL Block Cache Improvements
I'll call soon on a vote on this if nobody has remarks. Le jeudi 04 juin 2015 16:05:00, Even Rouault a écrit : Hi, I've updated an old RFC initiated by Tamas. The main idea, having a hashset based implementation as an alternative to the array based, remains. Changes consist mainly in code restructuration, perf improvements to reduce lock contention and porting to the state of the latest code base. This is a RFC for GDAL 2.1 Details at https://trac.osgeo.org/gdal/wiki/rfc26_blockcache == Summary and rationale == GDAL maintains an in-memory cache for the raster blocks fetched from the drivers and ensures that the second attempt to access the same block will be served from the cache instead of the driver. This cache is maintained in a per-band fashion and an array is allocated for the pointers for each blocks (or sub-blocks). This approach is not sufficient with large raster dimensions (or large virtual rasters ie. with the WMS/TMS driver), which may cause out of memory errors in GDALRasterBand::InitBlockInfo, as raised in #3224 For example, a band of a dataset at level 21 with a GoogleMaps tiling requires 2097152x2097152 tiles of 256x256 pixels. This means that GDAL will try to allocate an array of 32768x32768 = 1 billion elements (32768 = 2097152 / 64). The size of this array is 4 GB on a 32-bit build, so it cannot be allocated at all. And it is 8 GB on a 64-bit build (even if this is generally only virtual memory reservation but not actually allocation of physical pages of memory, due to over-commit mechanism of the operating system). At dataset closing, this means that those 1 billion cells will have to be explored to discover remaining cached blocks. In reality, all above figures must be multiplied by 3 for a RGB (or 4 for a RGBA) dataset. In the hash set implementation, memory allocation depends directly on the number of cached blocks. Typically with the default GDAL_CACHEMAX size of 40 MB, only 640 blocks of 256x256 pixels can be simultaneously cached (for all datasets). Best regards, Even -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] gdalwarp on a separate RPC model
Dear List, I have a question about recitify satellite images that has RPC camera using gdalwarp. I know gdalwarp can be used for this task, with using RPC metadata (either packed inside the NITF or stored inside RPB files) Normally, the image folder shall contain following files input.ntf input.rpb input.imd etc... The command I used for this is gdalwarp -rpc input.ntf -to RPC_HEIGHT=200 rectified.tif Now I performed some geo-correction on the satellite image and generate a corrected RPC model (with imporved offset values) and I store this correct RPC camera in 'correct.rpb'. Now I am trying to use this 'correct.rpb' to recitify the image I tried gdalwarp -rpc input.ntf correct.rpb -to RPC_HEIGHT=200 recitify_corrected.tif but it failed with following error ERROR 4: 'correct.rpb' not recognized as a supported file format Then I tried to overwrite the 'input.rpb' in the image folder with my 'corrected.rpb', run 'gdalwarp -rpc input.ntf -to RPC_HEIGHT=200 rectified.tif' again but I got exactly same results as I used the original RPC model. I guess that might be because gdalwarp used RPC parameter that was packed inside NITF image itself, instead of getting them from rpb files ? So Is there other way to use my corrected RPC model in gdal, like gdalwarp, gdal_transform, etc.? Also, gdaltransform doesn't not accept RPB files either. It only recognizes GDAL dataset, i.e. vrt file. Therefore I need to generate vrt file from my RPB. Is there any vrt example for RPC camera that I can follow? Thanks a lot Ian ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] gdalwarp on a separate RPC model
Hi Yi, GDAL use the RPC which it report by gdalinfo. If gdalinfo reports old RPC, you have to clean them from dataset. It seem's to me that rpb (*_rpc.txt, etc.) files accepted only on GeoTIFF and TILL formats. On GDAL 2.0 also jp2 added. So NITF is not a right format for use external RPC. So, first of all try to gdal_translate you format to GTiff, and than try to create so-name rpb file. Best regards, Dmitry 15.06.2015 21:18, Yi Dong пишет: Dear List, I have a question about recitify satellite images that has RPC camera using gdalwarp. I know gdalwarp can be used for this task, with using RPC metadata (either packed inside the NITF or stored inside RPB files) Normally, the image folder shall contain following files input.ntf input.rpb input.imd etc... The command I used for this is gdalwarp -rpc input.ntf -to RPC_HEIGHT=200 rectified.tif Now I performed some geo-correction on the satellite image and generate a corrected RPC model (with imporved offset values) and I store this correct RPC camera in 'correct.rpb'. Now I am trying to use this 'correct.rpb' to recitify the image I tried gdalwarp -rpc input.ntf correct.rpb -to RPC_HEIGHT=200 recitify_corrected.tif but it failed with following error ERROR 4: 'correct.rpb' not recognized as a supported file format Then I tried to overwrite the 'input.rpb' in the image folder with my 'corrected.rpb', run 'gdalwarp -rpc input.ntf -to RPC_HEIGHT=200 rectified.tif' again but I got exactly same results as I used the original RPC model. I guess that might be because gdalwarp used RPC parameter that was packed inside NITF image itself, instead of getting them from rpb files ? So Is there other way to use my corrected RPC model in gdal, like gdalwarp, gdal_transform, etc.? Also, gdaltransform doesn't not accept RPB files either. It only recognizes GDAL dataset, i.e. vrt file. Therefore I need to generate vrt file from my RPB. Is there any vrt example for RPC camera that I can follow? Thanks a lot Ian ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Gdal-Grid lidar.
Thanks Even, One last question about the buffer. Is 2GB is the max buffer size for gdal_grid? If I select that, am I likely to run into problem with some values? Apart from trial and error, is there a balance that should be stuck between available computer memory (currently at 64GB), input file size and the buffer or can I simply select a 2GB buffer? Merci beaucoup pour votre aide. Nicolas Cadieux M.Sc. Les Entreprises Archéotec inc. 8548, rue Saint-Denis Montréal H2P 2H2 Téléphone: 514.381.5112 Fax: 514.381.4995 www.archeotec.ca On Jun 12, 2015 17:29, Even Rouault even.roua...@spatialys.com wrote: Le vendredi 12 juin 2015 23:18:26, Nicolas Cadieux a écrit : Hi Even, Thank you very much. I increased the buffer to 1.5 GB (512+512+514)*1024*1024 and I can now work much much faster with gdal_grid. Does gdal_fillnodata have the same type of variable that could speed things up? It is also slow and does not seem speeded up by the All_CPUS. No, no tunable for fillnodata, and it doesn't support multi-threaded parallelization. I will use warp and the raster calculator in the future so you have more magic up your sleeve, I would appreciate it. Cheers and thanks again. Nicolas Cadieux M.Sc. Les Entreprises Archéotec inc. 8548, rue Saint-Denis Montréal H2P 2H2 Téléphone: 514.381.5112 Fax: 514.381.4995 www.archeotec.ca On Jun 5, 2015 19:13, Even Rouault even.roua...@spatialys.com wrote: Le samedi 06 juin 2015 00:18:45, Nicolas Cadieux a écrit : Hi, I have been using gdal_grid to interpolate LiDAR data for some time now. This would be a typical command: gdal_grid -l P0344_sol2004_XYZI_MTM8_CGVD28_150529 -txe 348733.0 401135.0 -tye 5568587.0 5773796.0 -a nearest:radius1=3.0:radius2=3.0:angle=0.0:nodata=0.0 -outsize 52402.0 205209.0 -of GTiff H:/Nicolas/pytemp/OutPut\P0344_sol2004_XYZI_MTM8_CGVD28_150529.vrt H:/Nicolas/pytemp/OutPut\P0344_sol2004_XYZI_MTM8_CGVD28_150529.tif --confi g GDAL_NUM_THREADS ALL_CPUS As you can see, this is a huge raster and the csv. file (Pointed with a .vrt) contains over a billion points. I am trying to do this in one shot to avoid boundary effects. (I know I can split this .csv file into smaller bits and make this more efficient). The computer I am using has 12 cores, 64BG of memory (smaller than the csv file) and a 1TB pcie solid state drive. It should manage the task but it has been running for 5 days with less than 10% done). I have access to a super computer with gdal if all else fails. This is my question. The --config GDAL_NUM_THREADS ALL_CPUS or --config GDAL_NUM_THREADS 12 does not seem to work with the nearest neighbor algorithm. (It seem to work with the -a count command) If I use this on a smaller data set, I see no difference in speed or cpu usage between using the ALL_CPUS or not using this switch. Is this a bug or simply a limitation of the nearest neighbor algorithm? Nicolas, The threading mechanism is generic for all algorithms . Add --debug on and you should see traces like GDAL_GRID: Using 12 threads. Now, a reason for it not being efficient is that the work buffer used by gdalgrid (16 MB) is too small w.r.t to nearest speed. Hum, actually looking more closely at the code, I see that the quad tree to index the points will be rebuilt for each work buffer. This is likely the main cause of the inefficiency and why you don't see measurable differences between non threaded or threaded computations (since the building of the quadtree is not multithreaded). If you compiled from source, you could try changing the const GUInt32 nDesiredBufferSize = 16*1024*1024; line in apps/gdal_grid.cpp to a much higher value. A more proper solution would be to avoid rebuilding the quad tree for each work buffer. This is mainly code restructuring. An alternative to gdal_grid would be to use gdal_rasterize + gdal_fillnodata. This is usually much much faster, so you could try to check if it gives the results you expect. You could also try scipy : http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.interpol ate.griddata.html#scipy.interpolate.griddata Not sure how it behaves with that number of points. Even I don't want to move this to a super computer if it will not multi-thread properly. Thanks, Nicolas Cadieux (Gdal1.11.2 downloaded with OSGEO4W 64bit install with QGIS 2.8 on windows 7_64) I use GDAL from the command line not from QGIS. Nicolas Cadieux M.Sc. Les Entreprises Archéotec inc. 8548, rue Saint-Denis Montréal H2P 2H2 Téléphone: 514.381.5112 Fax: 514.381.4995 www.archeotec.ca ___ gdal-dev mailing list
Re: [gdal-dev] gdalwarp on a separate RPC model
Le lundi 15 juin 2015 20:18:02, Yi Dong a écrit : Dear List, I have a question about recitify satellite images that has RPC camera using gdalwarp. I know gdalwarp can be used for this task, with using RPC metadata (either packed inside the NITF or stored inside RPB files) Normally, the image folder shall contain following files input.ntf input.rpb input.imd etc... The command I used for this is gdalwarp -rpc input.ntf -to RPC_HEIGHT=200 rectified.tif Now I performed some geo-correction on the satellite image and generate a corrected RPC model (with imporved offset values) and I store this correct RPC camera in 'correct.rpb'. Now I am trying to use this 'correct.rpb' to recitify the image I tried gdalwarp -rpc input.ntf correct.rpb -to RPC_HEIGHT=200 recitify_corrected.tif but it failed with following error ERROR 4: 'correct.rpb' not recognized as a supported file format Then I tried to overwrite the 'input.rpb' in the image folder with my 'corrected.rpb', run 'gdalwarp -rpc input.ntf -to RPC_HEIGHT=200 rectified.tif' again but I got exactly same results as I used the original RPC model. I guess that might be because gdalwarp used RPC parameter that was packed inside NITF image itself, instead of getting them from rpb files ? So Is there other way to use my corrected RPC model in gdal, like gdalwarp, gdal_transform, etc.? Also, gdaltransform doesn't not accept RPB files either. It only recognizes GDAL dataset, i.e. vrt file. Therefore I need to generate vrt file from my RPB. Is there any vrt example for RPC camera that I can follow? Ian, I'd suggesting gdal_translate'ing your input.ntf into a VRT, and replace the values in the VRT metadata domain with your own values. Then you can run gdalwarp on this VRT file Even -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Gdal-Grid lidar.
Hi Nicolas, I see someone already posted something about lidar2dems. We also wanted to use only open-source, so put together this project to easily be able to make DEMs using what's out there already. The documentation gives an overview of what it can do and a tutorial for processing a large area. http://applied-geosolutions.github.io/lidar2dems/ I'd like to add multiprocessing support, as it processes in pieces as it is, so wouldn't take much more work to split it up over processors. -matt On Mon, Jun 15, 2015 at 3:00 PM, Nicolas Cadieux nicolas.cadi...@archeotec.ca wrote: Thanks Even, One last question about the buffer. Is 2GB is the max buffer size for gdal_grid? If I select that, am I likely to run into problem with some values? Apart from trial and error, is there a balance that should be stuck between available computer memory (currently at 64GB), input file size and the buffer or can I simply select a 2GB buffer? Merci beaucoup pour votre aide. Nicolas Cadieux M.Sc. Les Entreprises Archéotec inc. 8548, rue Saint-Denis Montréal H2P 2H2 Téléphone: 514.381.5112 Fax: 514.381.4995 www.archeotec.ca On Jun 12, 2015 17:29, Even Rouault even.roua...@spatialys.com wrote: Le vendredi 12 juin 2015 23:18:26, Nicolas Cadieux a écrit : Hi Even, Thank you very much. I increased the buffer to 1.5 GB (512+512+514)*1024*1024 and I can now work much much faster with gdal_grid. Does gdal_fillnodata have the same type of variable that could speed things up? It is also slow and does not seem speeded up by the All_CPUS. No, no tunable for fillnodata, and it doesn't support multi-threaded parallelization. I will use warp and the raster calculator in the future so you have more magic up your sleeve, I would appreciate it. Cheers and thanks again. Nicolas Cadieux M.Sc. Les Entreprises Archéotec inc. 8548, rue Saint-Denis Montréal H2P 2H2 Téléphone: 514.381.5112 Fax: 514.381.4995 www.archeotec.ca On Jun 5, 2015 19:13, Even Rouault even.roua...@spatialys.com wrote: Le samedi 06 juin 2015 00:18:45, Nicolas Cadieux a écrit : Hi, I have been using gdal_grid to interpolate LiDAR data for some time now. This would be a typical command: gdal_grid -l P0344_sol2004_XYZI_MTM8_CGVD28_150529 -txe 348733.0 401135.0 -tye 5568587.0 5773796.0 -a nearest:radius1=3.0:radius2=3.0:angle=0.0:nodata=0.0 -outsize 52402.0 205209.0 -of GTiff H:/Nicolas/pytemp/OutPut\P0344_sol2004_XYZI_MTM8_CGVD28_150529.vrt H:/Nicolas/pytemp/OutPut\P0344_sol2004_XYZI_MTM8_CGVD28_150529.tif --confi g GDAL_NUM_THREADS ALL_CPUS As you can see, this is a huge raster and the csv. file (Pointed with a .vrt) contains over a billion points. I am trying to do this in one shot to avoid boundary effects. (I know I can split this .csv file into smaller bits and make this more efficient). The computer I am using has 12 cores, 64BG of memory (smaller than the csv file) and a 1TB pcie solid state drive. It should manage the task but it has been running for 5 days with less than 10% done). I have access to a super computer with gdal if all else fails. This is my question. The --config GDAL_NUM_THREADS ALL_CPUS or --config GDAL_NUM_THREADS 12 does not seem to work with the nearest neighbor algorithm. (It seem to work with the -a count command) If I use this on a smaller data set, I see no difference in speed or cpu usage between using the ALL_CPUS or not using this switch. Is this a bug or simply a limitation of the nearest neighbor algorithm? Nicolas, The threading mechanism is generic for all algorithms . Add --debug on and you should see traces like GDAL_GRID: Using 12 threads. Now, a reason for it not being efficient is that the work buffer used by gdalgrid (16 MB) is too small w.r.t to nearest speed. Hum, actually looking more closely at the code, I see that the quad tree to index the points will be rebuilt for each work buffer. This is likely the main cause of the inefficiency and why you don't see measurable differences between non threaded or threaded computations (since the building of the quadtree is not multithreaded). If you compiled from source, you could try changing the const GUInt32 nDesiredBufferSize = 16*1024*1024; line in apps/gdal_grid.cpp to a much higher value. A more proper solution would be to avoid rebuilding the quad tree for each work buffer. This is mainly code restructuring. An alternative to gdal_grid would be to use gdal_rasterize + gdal_fillnodata. This is usually much much faster, so you could try to check if it gives the results you expect. You could also try scipy : http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.interpol ate.griddata.html#scipy.interpolate.griddata Not
Re: [gdal-dev] gdalwarp on a separate RPC model
Hi Evan, Thanks a lot! Your suggestion is exactly what I am looking for. To keep the record, here is what I did 1. gdal_translate -of VRT input.ntf input.vrt 2. modify the 'input.vrt' w.r.t. correct RPC model, name it as 'input_correct.vrt' 3. gdalwarp -rpc -to RPC_HEIGHT=200 input_correct.vrt output.tif 4. gdaltransform input_correct.vrt for projection 5. gdaltransform -i input_correct.vrt for backprojection Thanks everyone again for help Ian On 6/15/2015 2:55 PM, Even Rouault wrote: Le lundi 15 juin 2015 20:18:02, Yi Dong a écrit : Dear List, I have a question about recitify satellite images that has RPC camera using gdalwarp. I know gdalwarp can be used for this task, with using RPC metadata (either packed inside the NITF or stored inside RPB files) Normally, the image folder shall contain following files input.ntf input.rpb input.imd etc... The command I used for this is gdalwarp -rpc input.ntf -to RPC_HEIGHT=200 rectified.tif Now I performed some geo-correction on the satellite image and generate a corrected RPC model (with imporved offset values) and I store this correct RPC camera in 'correct.rpb'. Now I am trying to use this 'correct.rpb' to recitify the image I tried gdalwarp -rpc input.ntf correct.rpb -to RPC_HEIGHT=200 recitify_corrected.tif but it failed with following error ERROR 4: 'correct.rpb' not recognized as a supported file format Then I tried to overwrite the 'input.rpb' in the image folder with my 'corrected.rpb', run 'gdalwarp -rpc input.ntf -to RPC_HEIGHT=200 rectified.tif' again but I got exactly same results as I used the original RPC model. I guess that might be because gdalwarp used RPC parameter that was packed inside NITF image itself, instead of getting them from rpb files ? So Is there other way to use my corrected RPC model in gdal, like gdalwarp, gdal_transform, etc.? Also, gdaltransform doesn't not accept RPB files either. It only recognizes GDAL dataset, i.e. vrt file. Therefore I need to generate vrt file from my RPB. Is there any vrt example for RPC camera that I can follow? Ian, I'd suggesting gdal_translate'ing your input.ntf into a VRT, and replace the values in the VRT metadata domain with your own values. Then you can run gdalwarp on this VRT file Even ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Gdal-Grid lidar.
Le lundi 15 juin 2015 21:00:15, Nicolas Cadieux a écrit : Thanks Even, One last question about the buffer. Is 2GB is the max buffer size for gdal_grid? If I select that, am I likely to run into problem with some values? The variable is a uint32 so you could probably go up to 4 GB. Apart from trial and error, is there a balance that should be stuck between available computer memory (currently at 64GB), input file size and the buffer or can I simply select a 2GB buffer? As you're exploring new territories, I'd expect you to report how this works ;-) Merci beaucoup pour votre aide. Nicolas Cadieux M.Sc. Les Entreprises Archéotec inc. 8548, rue Saint-Denis Montréal H2P 2H2 Téléphone: 514.381.5112 Fax: 514.381.4995 www.archeotec.ca On Jun 12, 2015 17:29, Even Rouault even.roua...@spatialys.com wrote: Le vendredi 12 juin 2015 23:18:26, Nicolas Cadieux a écrit : Hi Even, Thank you very much. I increased the buffer to 1.5 GB (512+512+514)*1024*1024 and I can now work much much faster with gdal_grid. Does gdal_fillnodata have the same type of variable that could speed things up? It is also slow and does not seem speeded up by the All_CPUS. No, no tunable for fillnodata, and it doesn't support multi-threaded parallelization. I will use warp and the raster calculator in the future so you have more magic up your sleeve, I would appreciate it. Cheers and thanks again. Nicolas Cadieux M.Sc. Les Entreprises Archéotec inc. 8548, rue Saint-Denis Montréal H2P 2H2 Téléphone: 514.381.5112 Fax: 514.381.4995 www.archeotec.ca On Jun 5, 2015 19:13, Even Rouault even.roua...@spatialys.com wrote: Le samedi 06 juin 2015 00:18:45, Nicolas Cadieux a écrit : Hi, I have been using gdal_grid to interpolate LiDAR data for some time now. This would be a typical command: gdal_grid -l P0344_sol2004_XYZI_MTM8_CGVD28_150529 -txe 348733.0 401135.0 -tye 5568587.0 5773796.0 -a nearest:radius1=3.0:radius2=3.0:angle=0.0:nodata=0.0 -outsize 52402.0 205209.0 -of GTiff H:/Nicolas/pytemp/OutPut\P0344_sol2004_XYZI_MTM8_CGVD28_150529.vrt H:/Nicolas/pytemp/OutPut\P0344_sol2004_XYZI_MTM8_CGVD28_150529.tif --confi g GDAL_NUM_THREADS ALL_CPUS As you can see, this is a huge raster and the csv. file (Pointed with a .vrt) contains over a billion points. I am trying to do this in one shot to avoid boundary effects. (I know I can split this .csv file into smaller bits and make this more efficient). The computer I am using has 12 cores, 64BG of memory (smaller than the csv file) and a 1TB pcie solid state drive. It should manage the task but it has been running for 5 days with less than 10% done). I have access to a super computer with gdal if all else fails. This is my question. The --config GDAL_NUM_THREADS ALL_CPUS or --config GDAL_NUM_THREADS 12 does not seem to work with the nearest neighbor algorithm. (It seem to work with the -a count command) If I use this on a smaller data set, I see no difference in speed or cpu usage between using the ALL_CPUS or not using this switch. Is this a bug or simply a limitation of the nearest neighbor algorithm? Nicolas, The threading mechanism is generic for all algorithms . Add --debug on and you should see traces like GDAL_GRID: Using 12 threads. Now, a reason for it not being efficient is that the work buffer used by gdalgrid (16 MB) is too small w.r.t to nearest speed. Hum, actually looking more closely at the code, I see that the quad tree to index the points will be rebuilt for each work buffer. This is likely the main cause of the inefficiency and why you don't see measurable differences between non threaded or threaded computations (since the building of the quadtree is not multithreaded). If you compiled from source, you could try changing the const GUInt32 nDesiredBufferSize = 16*1024*1024; line in apps/gdal_grid.cpp to a much higher value. A more proper solution would be to avoid rebuilding the quad tree for each work buffer. This is mainly code restructuring. An alternative to gdal_grid would be to use gdal_rasterize + gdal_fillnodata. This is usually much much faster, so you could try to check if it gives the results you expect. You could also try scipy : http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.inte rpol ate.griddata.html#scipy.interpolate.griddata Not sure how it behaves with that number of points. Even I don't want to move this to a super computer if it will not multi-thread properly. Thanks, Nicolas Cadieux (Gdal1.11.2 downloaded with OSGEO4W 64bit install with QGIS 2.8 on windows 7_64) I use GDAL from the command line not from QGIS. Nicolas
Re: [gdal-dev] GDAL/OGR 2.0.0 RC2 Available for Review
RC2 compiles and runs on Windows. -jeff On 2015-06-14 4:28 PM, Even Rouault wrote: Hi, Due to a regression that could cause compressed GeoTIFF created by GDAL to be almost twice as large as their optimal size in some circumstances (see https://trac.osgeo.org/gdal/ticket/5999), I have prepared a RC2. The list of fixes in the 2.0 branch since RC1: * Java GNUmakefile: add -f in rm commands (#5997) * Doc: Add a brief summary to the GDAL and OGR utilities pages (#5996) * Various typo fixes in doc and code (#5995) * GTiff/block cache: fix creation of pixel interleaved multiband files (2.0 regression, #5999) * VRT: add handling of a shared='0' attribute on SourceFilename to open sources in non-shared mode, and VRT_SHARED_SOURCE config option that can be set to 0 in case the shared attribute isn't there (#5992) * Ili: Fix crash with models using types derived from INTERLIS * gdal_grid: fix crash in optimized mode with GCC 4.4 on 64bit (#5987) * GDALSuggestedWarpOut2(): fix typo that could prevent detection of part of the error cases for #5980 * CPLAcquireMutex(): improve performance on Windows (#5986) Updated source code: http://download.osgeo.org/gdal/2.0.0/gdal-2.0.0RC2.tar.xz http://download.osgeo.org/gdal/2.0.0/gdal-2.0.0RC2.tar.gz http://download.osgeo.org/gdal/2.0.0/gdal200RC2.zip gdal-grass plugin unchanged: http://download.osgeo.org/gdal/2.0.0/gdal-grass-2.0.0RC1.tar.gz Updated snapshot of the gdalautotest suite: http://download.osgeo.org/gdal/2.0.0/gdalautotest-2.0.0RC2.tar.gz http://download.osgeo.org/gdal/2.0.0/gdalautotest-2.0.0RC2.zip Updated documentation: http://download.osgeo.org/gdal/2.0.0/gdal200docRC2.zip I'll call for a vote promoting it to final in one day if no serious problems are reported. Best regards, Even ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] gdalwarp on a separate RPC model
* Yi Dong yidon...@gmail.com [2015-06-15 14:18:02 -0400]: Dear List, I have a question about recitify satellite images that has RPC camera using gdalwarp. I know gdalwarp can be used for this task, with using RPC metadata (either packed inside the NITF or stored inside RPB files) Normally, the image folder shall contain following files input.ntf input.rpb input.imd etc... The command I used for this is gdalwarp -rpc input.ntf -to RPC_HEIGHT=200 rectified.tif Now I performed some geo-correction on the satellite image and generate a corrected RPC model (with imporved offset values) Deat Ian, may I ask how you generated the corrected RPCs after the geometric (I guess) correction(s)? Thank you, Nikos and I store this correct RPC camera in 'correct.rpb'. Now I am trying to use this 'correct.rpb' to recitify the image I tried gdalwarp -rpc input.ntf correct.rpb -to RPC_HEIGHT=200 recitify_corrected.tif but it failed with following error ERROR 4: 'correct.rpb' not recognized as a supported file format Then I tried to overwrite the 'input.rpb' in the image folder with my 'corrected.rpb', run 'gdalwarp -rpc input.ntf -to RPC_HEIGHT=200 rectified.tif' again but I got exactly same results as I used the original RPC model. I guess that might be because gdalwarp used RPC parameter that was packed inside NITF image itself, instead of getting them from rpb files ? So Is there other way to use my corrected RPC model in gdal, like gdalwarp, gdal_transform, etc.? Also, gdaltransform doesn't not accept RPB files either. It only recognizes GDAL dataset, i.e. vrt file. Therefore I need to generate vrt file from my RPB. Is there any vrt example for RPC camera that I can follow? Thanks a lot Ian ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev -- Nikos Alexandris | Remote Sensing Scientist, Dr Themidos 3, 42100, Trikala, Greece GPG Key Fingerprint 6F9D4506F3CA28380974D31A9053534B693C4FB3 ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Fwd: [SoC] Report 3 - Integration of GDAL Utilities into GDAL Core library
Great work, really looking forward to it. If i understand it correctly this should make working with GDAL much more robust if you are using it with language bindings. For example, when using Conda (package/env manager) with GDAL and the Python bindings. I often put the location of the GDAL utilities in my PATH, so calling `gdalwarp` will work from anywhere. But if you make a new environment with a different GDAL version, the PATH will still point to the old version, and you would suddenly mix versions together if you combine the utilities with the bindings. Also, not everyone ships the utilities when they compile GDAL. -- View this message in context: http://osgeo-org.1560.x6.nabble.com/gdal-dev-Fwd-SoC-Report-3-Integration-of-GDAL-Utilities-into-GDAL-Core-library-tp5210878p5210921.html Sent from the GDAL - Dev mailing list archive at Nabble.com. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Overlay shapefile onto Geotiff image
Graeme B. Bell wrote You'll need to tell it what size you want to burn. You can do that by looking at the extents (northeast, southwest) of your current raster/geotiff. If you make a copy of the Geotiff you already have, you can provide it as the 'destination dataset' for gdal_rasterize, it would then (of course) automatically pick up all the output properties for you. You can then for example first burn with value 1, and then to another pass with burn value 0 and set the '-i' flag to invert the rasterization, that way anything not touching the lines will be set to 0, which you can later use as a nodata value to get the transparency. Note that the destination dataset for gdal_rasterize gets modified 'in place', so especially when experimenting make sure you always have a copy/backup. -- View this message in context: http://osgeo-org.1560.x6.nabble.com/gdal-dev-Overlay-shapefile-onto-Geotiff-image-tp5209980p5210923.html Sent from the GDAL - Dev mailing list archive at Nabble.com. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
