Re: [gdal-dev] limiting drivers that get registered
On 27/03/18 22:00, Even Rouault wrote: > Ah that was apparently your actual need rather than limiting the number of > drivers per-se. Do you have profiled where time was lost if you find opening > too slow ? The identification logic of drivers is supposed to operate fast > (extension based, or on the content of the first kilobyte of the header), but > there might be some drivers where improvements can be done. I wrote a pathological benchmark that opened and closed an HDF5 file 10,000 times. Moving the HDF5 driver to the head of the driver list gave about a 5x speedup. Thanks for the idea earlier. I came up with an acceptable solution: register all drivers, look to see which ones are present by using GDALGetDriverShortName(), and then de-register all the drivers. Cheers, Ben ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] limiting drivers that get registered
On mardi 27 mars 2018 21:51:46 CEST Ben Elliston wrote: > On 27/03/18 21:49, Even Rouault wrote: > > Hum why would you need to call GDALAllRegister() if you already manually > > registered some of them ? > > Because reordering the driver list by placing commonly used drivers at > the head of the list makes a significant difference in the time taken to > open files. Ah that was apparently your actual need rather than limiting the number of drivers per-se. Do you have profiled where time was lost if you find opening too slow ? The identification logic of drivers is supposed to operate fast (extension based, or on the content of the first kilobyte of the header), but there might be some drivers where improvements can be done. -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] limiting drivers that get registered
On 27/03/18 21:49, Even Rouault wrote: > Hum why would you need to call GDALAllRegister() if you already manually > registered some of them ? Because reordering the driver list by placing commonly used drivers at the head of the list makes a significant difference in the time taken to open files. > Well, call GDALAllRegister() and then query GDALGetDriverByName(driver_name). > You can unregister a driver with GDALDeregisterDriver() Thanks -- you've given me an idea .. Cheers, Ben ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] limiting drivers that get registered
On mardi 27 mars 2018 21:09:41 CEST Ben Elliston wrote: > Hi Even > > On 08/03/18 23:34, Even Rouault wrote: > > If you use GDAL as a library from your own code (C/C++), you can also > > directly call the GDALRegister_() / OGRRegister() functions you > > are interested in, instead of GDALAllRegister() > > This is ideal, as it allows a handful of drivers to be registered first, > and then GDALAllRegister() to be called to register everything else down > the line Hum why would you need to call GDALAllRegister() if you already manually registered some of them ? > . Unfortunately, it is impossible to know which drivers are > present until you call an undefined function in the GDAL shared library. > For example, GDALRegister_HDF4() is not defined if the HDF4 library > isn't found at configure time. > > Is there any way to know which drivers are available at runtime? Well, call GDALAllRegister() and then query GDALGetDriverByName(driver_name). You can unregister a driver with GDALDeregisterDriver() Even -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] limiting drivers that get registered
Hi Even On 08/03/18 23:34, Even Rouault wrote: > If you use GDAL as a library from your own code (C/C++), you can also > directly call the GDALRegister_() / OGRRegister() functions you > are interested in, instead of GDALAllRegister() This is ideal, as it allows a handful of drivers to be registered first, and then GDALAllRegister() to be called to register everything else down the line. Unfortunately, it is impossible to know which drivers are present until you call an undefined function in the GDAL shared library. For example, GDALRegister_HDF4() is not defined if the HDF4 library isn't found at configure time. Is there any way to know which drivers are available at runtime? Cheers, Ben ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] limiting drivers that get registered
On 09/03/18 10:43, br...@frogmouth.net wrote: > I think you should feel free to make general improvements to the autotools > setup š Is this my past coming back to haunt me? :-) Ben ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] limiting drivers that get registered
I think you should feel free to make general improvements to the autotools setup š Brad -Original Message- From: gdal-dev On Behalf Of Ben Elliston Sent: Thursday, 8 March 2018 11:24 PM To: gdal-dev@lists.osgeo.org Subject: [gdal-dev] limiting drivers that get registered Is it possible to limit the number of drivers compiled into GDAL (I am guessing not)? Alternatively, would a patch be welcome to read a confguration file to limit which drivers are registered at initialisation? Cheers, Ben ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] limiting drivers that get registered
I use bazel for building, so the build info isn't useful. You'll have to hack the makefiles a bit. I also tried to stop things from registering... find . -name \*.cpp | egrep -v 'autotest|swig|apps|fuzzers|gdaldrivermanager.cpp|ogrregisterall.cpp|gdalallregister.cpp' | xargs egrep 'Register[(][)];|OGRRegisterAll|GDALRegister' | egrep -v 'void|: +[*]|:/' Here are my two diffs: (note that I don't define FRMT_raw in my builds) diff -u a/gdalallregister.cpp b/gdalallregister.cpp --- a/gdalallregister.cpp 2018-03-08 10:40:24.342388944 -0800 +++ b/gdalallregister.cpp 2017-12-28 10:13:13.0 -0800 @@ -67,7 +67,8 @@ #ifdef FRMT_vrt GDALRegister_VRT(); -GDALRegister_Derived(); +// GDALRegister_Derived(); #endif #ifdef FRMT_gtiff @@ -220,7 +221,8 @@ #endif #ifdef FRMT_netcdf -GDALRegister_GMT(); +// GDALRegister_GMT(); GDALRegister_netCDF(); #endif @@ -406,6 +408,11 @@ /* ASCII-header in the same directory. */ /* */ +GDALRegister_EHdr(); +GDALRegister_ENVI(); + #ifdef FRMT_raw GDALRegister_PNM(); GDALRegister_DOQ1(); @@ -476,7 +483,8 @@ #endif #ifdef FRMT_hdf5 -GDALRegister_BAG(); +// GDALRegister_BAG(); GDALRegister_HDF5(); GDALRegister_HDF5Image(); #endif diff -u a/ogrregisterall.cpp b/ogrregisterall.cpp --- a/ogrregisterall.cpp 2018-03-08 10:54:56.775492742 -0800 +++ b/ogrregisterall.cpp 2018-03-08 10:54:27.643856370 -0800 @@ -106,7 +106,7 @@ RegisterOGRGMT(); #endif #ifdef SQLITE_ENABLED -RegisterOGRGeoPackage(); +// RegisterOGRGeoPackage(); RegisterOGRSQLite(); #endif #ifdef DODS_ENABLED On Thu, Mar 8, 2018 at 9:21 AM, Tobias Wendorff < tobias.wendo...@tu-dortmund.de> wrote: > Am Do, 8.03.2018, 17:47 schrieb Kurt Schwehr: > > I get a little more aggressive compared to what others suggested by > > commenting out register calls in gdalallregister.cpp and > > ogrregisterall.cpp > > and removing the code for the drivers. > > > > Share, share, share :) > > -- -- http://schwehr.org ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] limiting drivers that get registered
Am Do, 8.03.2018, 17:47 schrieb Kurt Schwehr: > I get a little more aggressive compared to what others suggested by > commenting out register calls in gdalallregister.cpp and > ogrregisterall.cpp > and removing the code for the drivers. > Share, share, share :) ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] limiting drivers that get registered
I get a little more aggressive compared to what others suggested by commenting out register calls in gdalallregister.cpp and ogrregisterall.cpp and removing the code for the drivers. There are gotchyas lurking, but it definitely works. e.g. Enable HDF5, but block BAG Enable EHDR & ENVI, but leave out the rest of the raw drivers On Thu, Mar 8, 2018 at 4:36 AM, Tobias Wendorff < tobias.wendo...@tu-dortmund.de> wrote: > I have created a config for minimal raster and vector. > > I can uploaded later when I am at home. > > -- > Von einem iPhone gesendet und wird daher Fehler enthaltenā¦ > > Am 08.03.2018 um 13:23 schrieb Ben Elliston : > > > Is it possible to limit the number of drivers compiled into GDAL (I am > > guessing not)? Alternatively, would a patch be welcome to read a > > confguration file to limit which drivers are registered at > initialisation? > > > > Cheers, > > Ben > > ___ > > gdal-dev mailing list > > gdal-dev@lists.osgeo.org > > https://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > -- -- http://schwehr.org ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] limiting drivers that get registered
I have created a config for minimal raster and vector. I can uploaded later when I am at home. -- Von einem iPhone gesendet und wird daher Fehler enthaltenā¦ Am 08.03.2018 um 13:23 schrieb Ben Elliston : > Is it possible to limit the number of drivers compiled into GDAL (I am > guessing not)? Alternatively, would a patch be welcome to read a > confguration file to limit which drivers are registered at initialisation? > > Cheers, > Ben > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] limiting drivers that get registered
Ben, > Is it possible to limit the number of drivers compiled into GDAL (I am > guessing not)? You have some knobs with the various ./configure or nmake.opt switch, but yes generally drivers that don't require external dependencies will always be compiled in. Editing manually the makefiles can help creating a smaller GDAL build but that can be a non- obvious task because of inter-driver dependencies. There's also a wiki page about that https://trac.osgeo.org/gdal/wiki/BuildingOnUnixWithMinimizedDrivers (I can see that I created a patch for a minimal build 10 years ago. Likely doesn't apply out of the box :-) ) > Alternatively, would a patch be welcome to read a > confguration file to limit which drivers are registered at initialisation? If you use GDAL as a library from your own code (C/C++), you can also directly call the GDALRegister_() / OGRRegister() functions you are interested in, instead of GDALAllRegister() GDALOpenEx() also accepts a white-list of drivers to probe. You are probably aware of the GDAL_SKIP configuration option to blacklist drivers at run- time. That said perhaps a GDAL_DRIVERS configuration option to whitelist could have some use. Even -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] limiting drivers that get registered
Is it possible to limit the number of drivers compiled into GDAL (I am guessing not)? Alternatively, would a patch be welcome to read a confguration file to limit which drivers are registered at initialisation? Cheers, Ben ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
