[gdal-dev] Support of filemapping for fast rendering in QGIS...
Hi, I want improve the rendering of vector features in QGIS.
I have opened a new branch of code in the repository of
QGIS(https://github.com/qgis/QGIS/pull/927 //
https://github.com/ahuarte47/QGIS/tree/Issue_8725-OGR) and the drawing tests
give me ~3x faster that original code :-)
I am testing with huge shapefiles (500.000 records) and I have detect that the
OGR-shapefile provider would be optimized using filemapping (boost
http://www.boost.org/, or others...)
I wonder if there is any support for FileMapping in gdal or planned. My
intention would be to redefine the access to the files in this provider with a
new function with SAHooks using FileMapping. I provide, as explanation, an
unfinished implementation using boost with which greatly speeds up the read of
files for critical, massive and repetitive processes such as the painting in a
map.
On the other hand, this provider may also accelerate with some small changes
that are pending of review in the ticket
(http://trac.osgeo.org/gdal/ticket/5272) that I hope will be evaluated.
thank you very much!
/
// TEST BOOST - MEMORY MAPPING FILE
#include boost/filesystem.hpp
#include boost/iostreams/device/mapped_file.hpp
#define ROOT_LIB_PATH( __RELATIVE_PATH ) /TFS_OSGeo/SIT/SIG_SDK/Open
Source/boost-1.54.0#__RELATIVE_PATH
#if _DEBUG
#pragma comment(lib, ROOT_LIB_PATH(
/lib32-msvc-10.0/libboost_iostreams-vc100-mt-sgd-1_54.lib ))
#pragma comment(lib, ROOT_LIB_PATH(
/lib32-msvc-10.0/libboost_filesystem-vc100-mt-sgd-1_54.lib))
#else
#pragma comment(lib, ROOT_LIB_PATH(
/lib32-msvc-10.0/libboost_iostreams-vc100-mt-s-1_54.lib ))
#pragma comment(lib, ROOT_LIB_PATH(
/lib32-msvc-10.0/libboost_filesystem-vc100-mt-s-1_54.lib ))
#endif
//---
// Helper wrapper class for boost::mapped_file.
class mapped_file_manager
{
public:
mapped_file_manager() : file(NULL), length(0), tell(0)
{
}
~mapped_file_manager()
{
close();
}
public:
boost::iostreams::mapped_file* file;
uintmax_t length;
uintmax_t tell;
public:
bool open(const char *pszFilename, bool readOnly)
{
if (boost::filesystem::exists(pszFilename))
{
boost::iostreams::mapped_file* fileP = new
boost::iostreams::mapped_file();
fileP-open(pszFilename, readOnly ?
boost::iostreams::mapped_file_source::readwrite :
boost::iostreams::mapped_file_source::readwrite);
if (fileP-is_open())
{
length =
boost::filesystem::file_size(pszFilename);
file = fileP;
return true;
}
}
return false;
}
bool close()
{
if (file)
{
length = 0;
tell = 0;
file-close();
delete file;
file = NULL;
return true;
}
return false;
}
};
SAFile MMF_SHP_Open( const char *pszFilename, const char *pszAccess )
{
if (boost::filesystem::exists(pszFilename))
{
mapped_file_manager* manager = new mapped_file_manager();
if (manager-open(pszFilename, strstr(pszAccess, r)!=NULL))
{
#if _DEBUG
size_t size = manager-file-size();
#endif
return (SAFile)manager;
}
delete manager;
}
return NULL;
}
SAOffset MMF_SHP_Read( void *p, SAOffset size, SAOffset nmemb, SAFile file )
{
mapped_file_manager* manager = (mapped_file_manager*)file;
char* data = manager-file-data();
int memoyrSize = size*nmemb;
memcpy(p, data+manager-tell, memoyrSize);
manager-tell += memoyrSize;
return nmemb;
}
SAOffset MMF_SHP_Write( void *p, SAOffset size, SAOffset nmemb, SAFile file )
{
mapped_file_manager* manager = (mapped_file_manager*)file;
char* data = manager-file-data();
int memoyrSize = size*nmemb;
memcpy(data+manager-tell, p, memoyrSize);
manager-tell += memoyrSize;
return nmemb;
}
SAOffset MMF_SHP_Seek( SAFile file, SAOffset offset, int whence )
{
mapped_file_manager* manager = (mapped_file_manager*)file;
if (whence==SEEK_SET)
{
manager-tell = offset;
return 0;
}
else
if (whence==SEEK_END)
{
manager-tell
Re: [gdal-dev] Support of filemapping for fast rendering in QGIS...
Selon A Huarte ahuart...@yahoo.es: Hi, I want improve the rendering of vector features in QGIS. I have opened a new branch of code in the repository of QGIS(https://github.com/qgis/QGIS/pull/927 // https://github.com/ahuarte47/QGIS/tree/Issue_8725-OGR) and the drawing tests give me ~3x faster that original code :-) On which OS ? Is it the performance gain of QGIS + OGR changes combined ? What is the speed-up in OGR side only ? You could for example use the test_ogrsf utility as a potential benchmark. You have to build explicitely with make test_ogrsf / nmake /f makefile.vc test_ogrsf.exe, in the apps directory of GDAL. I am testing with huge shapefiles (500.000 records) and I have detect that the OGR-shapefile provider would be optimized using filemapping (boost http://www.boost.org/, or others...) I wonder if there is any support for FileMapping in gdal or planned. There is none currently. This is something that has been talked about by a few people, but never implemented yet. I actually had a local experiment on Linux but I couldn't observe any real performance gain by using mmap() rather than fread(). Things are maybe different on the Windows side. My intention would be to redefine the access to the files in this provider with a new function with SAHooks using FileMapping. I provide, as explanation, an unfinished implementation using boost with which greatly speeds up the read of files for critical, massive and repetitive processes such as the painting in a map. A more general approach that could benefit to other drivers than shapefile (potentially most GDAL and OGR drivers actually) would be to implement a /vsimemmapped/ virtual file system driver. You have many examples of such drivers in the port subdirectory. And like it is done currently with /vsicache/, the default virtual file system in cpl_vsil_unix_stdio_64.cpp or cpl_vsil_win32.cpp could be instructed to use /vsicache/ if an env. variable is toggled. On the other hand, this provider may also accelerate with some small changes that are pending of review in the ticket (http://trac.osgeo.org/gdal/ticket/5272) that I hope will be evaluated. I had skimmed through that one too : * Did you evaluate the precision loss of using OGRFastAtof() rather than CPLAtof() ? * Are you sure that the performance improvement of replacing memcpy() by *dest = *src is significant ? A decent compiler (GCC ;-)) should optimize that on most platforms anyway. The thing is that there are still people using odd platforms where a non-aligned pointer assingment will cause a SIGBUS, hence the need for memcpy(). If the performance gain is significant enough we could perhaps have a MEMCPY_DOUBLE macro that on X86 would evaluate to *dest = *ptr and on other platforms would default to memcpy(). Best regards, Even ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Rv: Support of filemapping for fast rendering in QGIS...
Hi, I am testing in Windows XP compiling with Visual Studio 2010.
I have a console test application using directly the shapefil library.
I will use test_ogrsf utility for a complete benchmark
I this environment and my test application, using OGRFastAtof(), the full read
of a huge shape around 500.000 records (500mb of shp file, 150mb of dbf file)
with eight double fields goes from 5.6sg to 3.06sg. I don't evaluate that the
precision loss, it is a task for check.
In addition, using filemapping with boost library, the full process goes from
3.06sg to 1.45sg.
These values as you see are approximate, but they give an idea of
optimization is possible at least for this OS.
It is why I wanted to check this behavior
thanks!/
// TEST BOOST - MEMORY MAPPING FILE
#include boost/filesystem.hpp
#include boost/iostreams/device/mapped_file.hpp
#define ROOT_LIB_PATH( __RELATIVE_PATH ) /TFS_OSGeo/SIT/SIG_SDK/Open
Source/boost-1.54.0#__RELATIVE_PATH
#if _DEBUG
#pragma comment(lib, ROOT_LIB_PATH(
/lib32-msvc-10.0/libboost_iostreams-vc100-mt-sgd-1_54.lib ))
#pragma comment(lib, ROOT_LIB_PATH(
/lib32-msvc-10.0/libboost_filesystem-vc100-mt-sgd-1_54.lib))
#else
#pragma comment(lib, ROOT_LIB_PATH(
/lib32-msvc-10.0/libboost_iostreams-vc100-mt-s-1_54.lib ))
#pragma comment(lib, ROOT_LIB_PATH(
/lib32-msvc-10.0/libboost_filesystem-vc100-mt-s-1_54.lib ))
#endif
//---
// Helper wrapper class for boost::mapped_file.
class mapped_file_manager
{
public:
mapped_file_manager() : file(NULL), length(0), tell(0)
{
}
~mapped_file_manager()
{
close();
}
public:
boost::iostreams::mapped_file* file;
uintmax_t length;
uintmax_t tell;
public:
bool open(const char *pszFilename, bool readOnly)
{
if (boost::filesystem::exists(pszFilename))
{
boost::iostreams::mapped_file* fileP = new
boost::iostreams::mapped_file();
fileP-open(pszFilename, readOnly ?
boost::iostreams::mapped_file_source::readwrite :
boost::iostreams::mapped_file_source::readwrite);
if (fileP-is_open())
{
length =
boost::filesystem::file_size(pszFilename);
file = fileP;
return true;
}
}
return false;
}
bool close()
{
if (file)
{
length = 0;
tell = 0;
file-close();
delete file;
file = NULL;
return true;
}
return false;
}
};
SAFile MMF_SHP_Open( const char *pszFilename, const char *pszAccess )
{
if (boost::filesystem::exists(pszFilename))
{
mapped_file_manager* manager = new mapped_file_manager();
if (manager-open(pszFilename, strstr(pszAccess, r)!=NULL))
{
#if _DEBUG
size_t size = manager-file-size();
#endif
return (SAFile)manager;
}
delete manager;
}
return NULL;
}
SAOffset MMF_SHP_Read( void *p, SAOffset size, SAOffset nmemb, SAFile file )
{
mapped_file_manager* manager = (mapped_file_manager*)file;
char* data = manager-file-data();
int memoyrSize = size*nmemb;
memcpy(p, data+manager-tell, memoyrSize);
manager-tell += memoyrSize;
return nmemb;
}
SAOffset MMF_SHP_Write( void *p, SAOffset size, SAOffset nmemb, SAFile file )
{
mapped_file_manager* manager = (mapped_file_manager*)file;
char* data = manager-file-data();
int memoyrSize = size*nmemb;
memcpy(data+manager-tell, p, memoyrSize);
manager-tell += memoyrSize;
return nmemb;
}
SAOffset MMF_SHP_Seek( SAFile file, SAOffset offset, int whence )
{
mapped_file_manager* manager = (mapped_file_manager*)file;
if (whence==SEEK_SET)
{
manager-tell = offset;
return 0;
}
else
if (whence==SEEK_END)
{
manager-tell = manager-length+offset;
return offset0 ? 0 : 1;
}
else
if (whence==SEEK_CUR)
{
manager-tell = manager-tell+offset;
return 0;
}
return 2;
}
SAOffset MMF_SHP_Tell( SAFile file )
{
mapped_file_manager* manager =
Re: [gdal-dev] Support of filemapping for fast rendering in QGIS...
Hi, I am testing in Windows XP compiling with Visual Studio 2010.
I have a console test application using directly the shapefil library.
I will use test_ogrsf utility for a complete benchmark
I this environment and my test application, using OGRFastAtof(), the full read
of a huge shape around 500.000 records (500mb of shp file, 150mb of dbf file)
with eight double fields goes from 5.6sg to 3.06sg. I don't evaluate that the
precision loss, it is a task for check.
In addition, using filemapping with boost library, the full process goes from
3.06sg to 1.45sg.
These values as you see are approximate, but they give an idea of
optimization is possible at least for this OS.
It is why I wanted to check this behavior
thanks!/
// TEST BOOST - MEMORY MAPPING FILE
#include boost/filesystem.hpp
#include boost/iostreams/device/mapped_file.hpp
#define ROOT_LIB_PATH( __RELATIVE_PATH ) /TFS_OSGeo/SIT/SIG_SDK/Open
Source/boost-1.54.0#__RELATIVE_PATH
#if _DEBUG
#pragma comment(lib, ROOT_LIB_PATH(
/lib32-msvc-10.0/libboost_iostreams-vc100-mt-sgd-1_54.lib ))
#pragma comment(lib, ROOT_LIB_PATH(
/lib32-msvc-10.0/libboost_filesystem-vc100-mt-sgd-1_54.lib))
#else
#pragma comment(lib, ROOT_LIB_PATH(
/lib32-msvc-10.0/libboost_iostreams-vc100-mt-s-1_54.lib ))
#pragma comment(lib, ROOT_LIB_PATH(
/lib32-msvc-10.0/libboost_filesystem-vc100-mt-s-1_54.lib ))
#endif
//---
// Helper wrapper class for boost::mapped_file.
class mapped_file_manager
{
public:
mapped_file_manager() : file(NULL), length(0), tell(0)
{
}
~mapped_file_manager()
{
close();
}
public:
boost::iostreams::mapped_file* file;
uintmax_t length;
uintmax_t tell;
public:
bool open(const char *pszFilename, bool readOnly)
{
if (boost::filesystem::exists(pszFilename))
{
boost::iostreams::mapped_file* fileP = new
boost::iostreams::mapped_file();
fileP-open(pszFilename, readOnly ?
boost::iostreams::mapped_file_source::readwrite :
boost::iostreams::mapped_file_source::readwrite);
if (fileP-is_open())
{
length =
boost::filesystem::file_size(pszFilename);
file = fileP;
return true;
}
}
return false;
}
bool close()
{
if (file)
{
length = 0;
tell = 0;
file-close();
delete file;
file = NULL;
return true;
}
return false;
}
};
SAFile MMF_SHP_Open( const char *pszFilename, const char *pszAccess )
{
if (boost::filesystem::exists(pszFilename))
{
mapped_file_manager* manager = new mapped_file_manager();
if (manager-open(pszFilename, strstr(pszAccess, r)!=NULL))
{
#if _DEBUG
size_t size = manager-file-size();
#endif
return (SAFile)manager;
}
delete manager;
}
return NULL;
}
SAOffset MMF_SHP_Read( void *p, SAOffset size, SAOffset nmemb, SAFile file )
{
mapped_file_manager* manager = (mapped_file_manager*)file;
char* data = manager-file-data();
int memoyrSize = size*nmemb;
memcpy(p, data+manager-tell, memoyrSize);
manager-tell += memoyrSize;
return nmemb;
}
SAOffset MMF_SHP_Write( void *p, SAOffset size, SAOffset nmemb, SAFile file )
{
mapped_file_manager* manager = (mapped_file_manager*)file;
char* data = manager-file-data();
int memoyrSize = size*nmemb;
memcpy(data+manager-tell, p, memoyrSize);
manager-tell += memoyrSize;
return nmemb;
}
SAOffset MMF_SHP_Seek( SAFile file, SAOffset offset, int whence )
{
mapped_file_manager* manager = (mapped_file_manager*)file;
if (whence==SEEK_SET)
{
manager-tell = offset;
return 0;
}
else
if (whence==SEEK_END)
{
manager-tell = manager-length+offset;
return offset0 ? 0 : 1;
}
else
if (whence==SEEK_CUR)
{
manager-tell = manager-tell+offset;
return 0;
}
return 2;
}
SAOffset MMF_SHP_Tell( SAFile file )
{
mapped_file_manager* manager =
[gdal-dev] Open BIL, BIP, and BSQ raster files without header
Dear all, I'm using GDAL 1.9.1 and I would like to load binary raster files especifying manually header information such number of lines, number of pixels, pixel data type, byte order, bits per pixel. Also I would like to specify file offset, band offset and line offset for the file. Can someone please help me figure this out. Thanks in advance, Regards -- Gabriel Fusca SUR Emprendimientos Tecnológicos Perú 345 Piso 5to Oficina B (C1067AAG) Ciudad de Buenos Aires, Argentina Tel. +54 (11) 4342-2976/84 gabrielfu...@suremptec.com.ar http://www.suremptec.com/ ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Postgis Raster issue
Hello,
I am playing with QGIS and raster date in postgis. I get somme error which get
me going to frmts/postgisraster/postgisrasterrasterband.cpp:469.
From QGIS I get Warning: RasterIO error: Error retrieving raster data from
database error message which come from GDAL.
If I read the postgresql log I get:
2013-10-22 17:38:47 CEST STATEMENT: SELECT st_band(rast, 1), st_width(rast),
st_height(rast), st_bandpixeltype(rast, 1), st_bandnodatavalue(rast, 1),
st_scalex(rast), st_scaley(rast), st_upperleftx(rast), st_upperlefty(rast)
FROM raster.mnt250 WHERE st_intersects(rast,
st_polygonfromtext('POLYGON((89875,0
7120125,0, 4789875,0
7120125,0, 4789875,0
2720125,0, 89875,0 2720125,0,
89875,0 7120125,0))', 2154))
Which is incorrect (check the decimal caracter) and is create on l.469 in the
cpp file.
My gdal is quiet old and is a 1.10dev (not 1.10 stable). As I can't build
again 1.10 stable or trunk, I wondering if it was something known.
Thanks,
Y.
PS : I tried to build gdal 1.10 and master and get this issue:
make[2]: entrant dans le répertoire «
/home/yjacolin/Documents/Dev/gdal/1.10/gdal/frmts/ogdi »
/bin/bash /home/yjacolin/Documents/Dev/gdal/1.10/gdal/libtool --mode=compile
--tag=CXX g++ -g -O2 -msse -DHAVE_SSE_AT_COMPILE_TIME -Wall -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/port -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/gcore -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/alg -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/ogr -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/ogr/ogrsf_frmts -
I/usr/include/ogdi -DOGR_ENABLED -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/port -c -o ../o/ogdidataset.lo
ogdidataset.cpp
libtool: compile: g++ -g -O2 -msse -DHAVE_SSE_AT_COMPILE_TIME -Wall -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/port -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/gcore -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/alg -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/ogr -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/ogr/ogrsf_frmts -
I/usr/include/ogdi -DOGR_ENABLED -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/port -c ogdidataset.cpp -fPIC -
DPIC -o ../o/.libs/ogdidataset.o
In file included from /usr/include/ogdi/ecs.h:353:0,
from ogdidataset.cpp:32:
/usr/include/ogdi/ecs_util.h:108:22: fatal error: projects.h: No such file or
directory
compilation terminated.
make[2]: *** [../o/ogdidataset.lo] Erreur 1
make[2]: quittant le répertoire «
/home/yjacolin/Documents/Dev/gdal/1.10/gdal/frmts/ogdi »
make[1]: *** [ogdi-install-obj] Erreur 2
make[1]: quittant le répertoire «
/home/yjacolin/Documents/Dev/gdal/1.10/gdal/frmts »
make: *** [frmts-target] Erreur 2
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Building HDF5 driver against the binary distribution
Hi Joaquim, Here it says: The SZIP and ZLIB external libraries are optional for use with HDF5. The HDF5 pre-compiled binary distributions include the SZIP (Encoder Enabled) and ZLIB libraries that they are compiled with. [http://www.hdfgroup.org/HDF5/release/obtain5.html] When they say include it means that the szip.dll and zlib.dll are there on the \HDF5\1.8.11\bin when you install the package. Did you compiled HDF5 1.8.11 with static szip and zlib? I build GDAL with the internal zlib. What about you? Thanks a lot, Ivan Date: Fri, 18 Oct 2013 23:31:58 +0100 From: jl...@ualg.pt To: even.roua...@mines-paris.org CC: gdal-dev@lists.osgeo.org Subject: Re: [gdal-dev] Building HDF5 driver against the binary distribution Hum, this reminds me very much of the very recent thread : http://lists.osgeo.org/pipermail/gdal-dev/2013-October/037257.html . So there might indeed be an issue with HDF5 1.8.11, and not specifically related to Windows. I didn't try myself. Did you try with an older Windows build provided by the HDF_Group ? (provided that you can download older builds...) Hi, I have build 1.8.11 myself and use it both in Mirone and GMT and had no problems so far. It's linked as dll, not a plugin but it should not matter. Joaquim ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Open BIL, BIP, and BSQ raster files without header
I'm thinking to configure the header file (.hdr) following the especification of the header ( http://webhelp.esri.com/arcgisdesktop/9.2/index.cfm?TopicName=BIL,_BIP,_and_BSQ_raster_files ) skipbytes: file offset bandrowbytes: band offset totalrowbytes: row offset I don't know if GDAL can handle this configuration. 2013/10/22 Kyle Shannon k...@pobox.com Gabriel, Did you look at the VRT dataset? http://www.gdal.org/gdal_vrttut.html I suspect that is what you are looking for. kss On Tue, Oct 22, 2013 at 7:27 AM, Gabriel Fusca gabrielfu...@suremptec.com.ar wrote: Dear all, I'm using GDAL 1.9.1 and I would like to load binary raster files especifying manually header information such number of lines, number of pixels, pixel data type, byte order, bits per pixel. Also I would like to specify file offset, band offset and line offset for the file. Can someone please help me figure this out. Thanks in advance, Regards -- Gabriel Fusca SUR Emprendimientos Tecnológicos Perú 345 Piso 5to Oficina B (C1067AAG) Ciudad de Buenos Aires, Argentina Tel. +54 (11) 4342-2976/84 gabrielfu...@suremptec.com.ar http://www.suremptec.com/ ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev -- Kyle -- Gabriel Fusca SUR Emprendimientos Tecnológicos Perú 345 Piso 5to Oficina B (C1067AAG) Ciudad de Buenos Aires, Argentina Tel. +54 (11) 4342-2976/84 gabrielfu...@suremptec.com.ar http://www.suremptec.com/ ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Postgis Raster issue
Le mardi 22 octobre 2013 17:53:27, Yves Jacolin (Free) a écrit :
Hello,
I am playing with QGIS and raster date in postgis. I get somme error which
get me going to frmts/postgisraster/postgisrasterrasterband.cpp:469.
From QGIS I get Warning: RasterIO error: Error retrieving raster data from
database error message which come from GDAL.
If I read the postgresql log I get:
2013-10-22 17:38:47 CEST STATEMENT: SELECT st_band(rast, 1),
st_width(rast), st_height(rast), st_bandpixeltype(rast, 1),
st_bandnodatavalue(rast, 1), st_scalex(rast), st_scaley(rast),
st_upperleftx(rast), st_upperlefty(rast) FROM raster.mnt250 WHERE
st_intersects(rast,
st_polygonfromtext('POLYGON((89875,0
7120125,0, 4789875,0
7120125,0, 4789875,0
2720125,0, 89875,0
2720125,0, 89875,0
7120125,0))', 2154))
Which is incorrect (check the decimal caracter) and is create on l.469 in
the cpp file.
Salut Yves,
Yes this is a known issue that has been fixed in trunk. This is due to the fact
that the locale within QGIS switches from the C locale to the locale of your
language that has comma as decimal separator. This would not be terribly
difficult to fix in 1.10 branch if needed. As a workaround, you can launch qgis
with LC_ALL=C qgis (but that will be in English)
My gdal is quiet old and is a 1.10dev (not 1.10 stable). As I can't build
again 1.10 stable or trunk, I wondering if it was something known.
Thanks,
Y.
PS : I tried to build gdal 1.10 and master and get this issue:
make[2]: entrant dans le répertoire «
/home/yjacolin/Documents/Dev/gdal/1.10/gdal/frmts/ogdi »
/bin/bash /home/yjacolin/Documents/Dev/gdal/1.10/gdal/libtool
--mode=compile --tag=CXX g++ -g -O2 -msse -DHAVE_SSE_AT_COMPILE_TIME
-Wall -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/port -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/gcore -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/alg -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/ogr -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/ogr/ogrsf_frmts -
I/usr/include/ogdi -DOGR_ENABLED -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/port -c -o
../o/ogdidataset.lo ogdidataset.cpp
libtool: compile: g++ -g -O2 -msse -DHAVE_SSE_AT_COMPILE_TIME -Wall -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/port -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/gcore -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/alg -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/ogr -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/ogr/ogrsf_frmts -
I/usr/include/ogdi -DOGR_ENABLED -
I/home/yjacolin/Documents/Dev/gdal/1.10/gdal/port -c ogdidataset.cpp -fPIC
- DPIC -o ../o/.libs/ogdidataset.o
In file included from /usr/include/ogdi/ecs.h:353:0,
from ogdidataset.cpp:32:
/usr/include/ogdi/ecs_util.h:108:22: fatal error: projects.h: No such file
or directory
compilation terminated.
make[2]: *** [../o/ogdidataset.lo] Erreur 1
make[2]: quittant le répertoire «
/home/yjacolin/Documents/Dev/gdal/1.10/gdal/frmts/ogdi »
make[1]: *** [ogdi-install-obj] Erreur 2
make[1]: quittant le répertoire «
/home/yjacolin/Documents/Dev/gdal/1.10/gdal/frmts »
make: *** [frmts-target] Erreur 2
Yes, known issue: latest Proj.4 version doesn't install projects.h any longer,
which was supposed to be a private header file, but is still needed by OGDI. If
you don't need OGDI, ./configure --without-ogdi . It should also be possible to
manually install projects.h in /usr/include;
Even
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
--
Geospatial professional services
http://even.rouault.free.fr/services.html
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Support of filemapping for fast rendering in QGIS...
On the other hand, this provider may also accelerate with some small changes that are pending of review in the ticket (http://trac.osgeo.org/gdal/ticket/5272) that I hope will be evaluated. Actually when thinking about the atof() optimization, a far better idea would be to make sure that QGIS uses OGR_L_SetIgnoredFields() with all the regular field names as parameter (except the fields it might use for styling), when it just needs the geometry. So no slow or fast atof() at all will be needed. In my old copy of QGis git, I see a QgsOgrProvider::setRelevantFields() method that is supposed to do something in that area. Something that you should be investigated more. You don't need to read 8 double fields just for rendering. Even -- Geospatial professional services http://even.rouault.free.fr/services.html ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Open BIL, BIP, and BSQ raster files without header
Le mardi 22 octobre 2013 19:43:50, Gabriel Fusca a écrit : I'm thinking to configure the header file (.hdr) following the especification of the header ( http://webhelp.esri.com/arcgisdesktop/9.2/index.cfm?TopicName=BIL,_BIP,_and _BSQ_raster_files ) skipbytes: file offset bandrowbytes: band offset totalrowbytes: row offset I don't know if GDAL can handle this configuration. This is handled by the GDAL EHdr driver. Looking at the code : * skipbytes: yes, taken into account * bandrowbytes: only taken into account if nbits 8 * totalrowbytes : only taken into account if nbits 8 Not sure why the last 2 fields are only taken into account conditionnaly. If that doesn't work for you, as suggested by Kyle, you could use VRTRawRasterBand specification. Even -- Geospatial professional services http://even.rouault.free.fr/services.html ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Open BIL, BIP, and BSQ raster files without header
I will try first the EHdr driver option. Thanks a lot Even and Kyle, 2013/10/22 Even Rouault even.roua...@mines-paris.org Le mardi 22 octobre 2013 19:43:50, Gabriel Fusca a écrit : I'm thinking to configure the header file (.hdr) following the especification of the header ( http://webhelp.esri.com/arcgisdesktop/9.2/index.cfm?TopicName=BIL,_BIP,_and _BSQ_raster_files ) skipbytes: file offset bandrowbytes: band offset totalrowbytes: row offset I don't know if GDAL can handle this configuration. This is handled by the GDAL EHdr driver. Looking at the code : * skipbytes: yes, taken into account * bandrowbytes: only taken into account if nbits 8 * totalrowbytes : only taken into account if nbits 8 Not sure why the last 2 fields are only taken into account conditionnaly. If that doesn't work for you, as suggested by Kyle, you could use VRTRawRasterBand specification. Even -- Geospatial professional services http://even.rouault.free.fr/services.html -- Gabriel Fusca SUR Emprendimientos Tecnológicos Perú 345 Piso 5to Oficina B (C1067AAG) Ciudad de Buenos Aires, Argentina Tel. +54 (11) 4342-2976/84 gabrielfu...@suremptec.com.ar http://www.suremptec.com/ ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Building HDF5 driver against the binary distribution
Hi Ivan I build HDF with zlib only (no szip) and dynamically but I don't understand your question about zlib and GDAL. I don't link GDAL with zlib Joaquim Hi Joaquim, Here it says: The SZIP and ZLIB external libraries are optional for use with HDF5. The HDF5 pre-compiled binary distributions include the SZIP (Encoder Enabled) and ZLIB libraries that they are compiled with. [http://www.hdfgroup.org/HDF5/release/obtain5.html] When they say include it means that the szip.dll and zlib.dll are there on the \HDF5\1.8.11\bin when you install the package. Did you compiled HDF5 1.8.11 with static szip and zlib? I build GDAL with the internal zlib. What about you? Thanks a lot, Ivan Date: Fri, 18 Oct 2013 23:31:58 +0100 From: jl...@ualg.pt To: even.roua...@mines-paris.org CC: gdal-dev@lists.osgeo.org Subject: Re: [gdal-dev] Building HDF5 driver against the binary distribution Hum, this reminds me very much of the very recent thread : http://lists.osgeo.org/pipermail/gdal-dev/2013-October/037257.html . So there might indeed be an issue with HDF5 1.8.11, and not specifically related to Windows. I didn't try myself. Did you try with an older Windows build provided by the HDF_Group ? (provided that you can download older builds...) Hi, I have build 1.8.11 myself and use it both in Mirone and GMT and had no problems so far. It's linked as dll, not a plugin but it should not matter. Joaquim ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Using a MapQuest TMS via GDAL
I'm trying to access a web mapping service (MapQuest in this case) using
the GDAL WMS driver.
Inspired by this page:
http://www.3liz.com/blog/rldhont/index.php?post/2012/07/17/OpenStreetMap-Tiles-in-QGIS
and this
http://developer.mapquest.com/web/products/open/map
I formulate an XML string such as this:
GDAL_WMSService name=TMSServerUrl
http://otile1.mqcdn.com/tiles/1.0.0/sat/${z}/${x}/${y}.png/ServerUrl/ServiceDataWindowUpperLeftX-1.035240e+007/UpperLeftXUpperLeftY5.297933e+006/UpperLeftYLowerRightX-1.035144e+007/LowerRightXLowerRightY5.297055e+006/LowerRightYTileLevel1/TileLevelTileCountX1/TileCountXTileCountY1/TileCountYYOrigintop/YOrigin/DataWindowProjectionEPSG:3857/ProjectionBlockSizeX256/BlockSizeXBlockSizeY256/BlockSizeYBandsCount3/BandsCountCache
//GDAL_WMS
I was hoping this was going to return a georeferenced raster of just the
area I'm interested in (defined by the DataWindow) but instead it seem to
return a virtual image comprising the whole world. However, it comes
bearing georeferencing info
Corner Coordinates:
Upper Left(-10352400.000, 5297933.000)
Lower Left(-10352400.000, 5297055.000)
Upper Right (-10351440.000, 5297933.000)
Lower Right (-10351440.000, 5297055.000)
Center(-10351920.000, 5297494.000)
That indicates it IS the area I requested. But when you look at it, it's
the whole world image.
Is the DataWindow just ignored here?
What is the proper process for requesting a DataWindow region and getting
a proper reoreferenced raster subset back?
And yes, we are within the license term for MapQuest. They've been
helping us, but it looks like this issue isn't really them, it's how I'm
using GDAL.
Google, VirtualEarth and OSM all react the same way -- ignoring the
DataWindow.
--
Chris 'Xenon' Hanson, omo sanza lettere. xe...@alphapixel.com
http://www.alphapixel.com/
Training • Consulting • Contracting
3D • Scene Graphs (Open Scene Graph/OSG) • OpenGL 2 • OpenGL 3 • OpenGL 4 •
GLSL • OpenGL ES 1 • OpenGL ES 2 • OpenCL
Digital Imaging • GIS • GPS • osgEarth • Terrain • Telemetry • Cryptography
• Digital Audio • LIDAR • Kinect • Embedded • Mobile • iPhone/iPad/iOS •
Android
@alphapixel https://twitter.com/alphapixel facebook.com/alphapixel (775)
623-PIXL [7495]
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Using a MapQuest TMS via GDAL
Chris Hanson xenon at alphapixel.com writes: What is the proper process for requesting a DataWindow region and getting a proper reoreferenced raster subset back? I am not sure about how changing the DataWindow size in the XML configuration file should work but anyway, it is not so good place to play. Instead, define the TMS source to cover the whole world as in http://www.gdal.org/frmt_wms_openstreetmap_tms.xml and control the area you want with the gdal_translate parameters http://www.gdal.org/gdal_translate.html I believe you want to use -projwin. -Jukka Rahkonen- ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Using a MapQuest TMS via GDAL
Is the DataWindow just ignored here? I am pretty sure the data window is actually interpreted as the tile extents of the service for the TMS mini driver. You'll notice that nothing else in the parameters could be used to georeference the tiles. I agree that window in this case is misleading and it has caused me confusion as well. I assume it was done that way so that the raster's pyramid/quad-tree is aligned with the service's zoom levels. For non-tiled services (e.g. WMS), the data window is probably what you expect it to be. I agree that warping/translating the data to the desired window is probably your best option, André On 2013-10-22 17:21, Jukka Rahkonen wrote: Chris Hanson xenon at alphapixel.com writes: What is the proper process for requesting a DataWindow region and getting a proper reoreferenced raster subset back? I am not sure about how changing the DataWindow size in the XML configuration file should work but anyway, it is not so good place to play. Instead, define the TMS source to cover the whole world as in http://www.gdal.org/frmt_wms_openstreetmap_tms.xml and control the area you want with the gdal_translate parameters http://www.gdal.org/gdal_translate.html I believe you want to use -projwin. -Jukka Rahkonen- ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
