Re: [gdal-dev] Speeding up gdalwarp process
We've been using Geotrellis (https://geotrellis.io/) for Spark jobs. -marius From: gdal-dev on behalf of Simon Sent: Thursday, January 23, 2020 6:30 PM To: gdal-dev@lists.osgeo.org Subject: [gdal-dev] Speeding up gdalwarp process Hi gdal-devs, I have a question, if there is some way to use gdalwarp in a clustering system (e.g., Sparks) or with GPU to speed up the process? My task involves re-projection and re-sampling of hundreds of high-resolution images. Any ideas to make use of Sparks or GPU is welcomed. Thank you. Simon ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Geopackages and Datetime fields
Richard, > I'm experimenting with writing time-based model results into > geopackages, and make those results (1 run = 1 geopackage) automatically > available as WMS-t. > > -1- First thing I wonder is how gdal/mapserver is handling geopackages > in a multi-threaded environment: if more web=clients are making a lot of > requests, is gdal/mapserver able to open the geopackage in parallel? > Reason I ask is because of the '-shm' and '-wal' files. As long as you don't write at the same time as you read, then no need to enable WAL mode. Concurrent reads on SQLite databases work out of the box. This is the SHARED locking mode of https://www.sqlite.org/lockingv3.html > > -2- Seconds Question is about Datetime handling. > Reading [0] I understand that SQLite (and so gpkg) does not have a > native store format for date/times. True for plain SQLite, but GPKG specifies a DATETIME time: See Table 1 at http://www.geopackage.org/spec121/#_sqlite_container """ DATETIME ISO-8601 date/time string in the form -MM-DDTHH:MM:SS.SSSZ with T separator character and Z suffix for coordinated universal time (UTC) encoded in either UTF-8 or UTF-16. See TEXT. Stored as SQLite TEXT. """ GDAL implements that. If putting other type of content in such columns, GDAL master since a couple days will warn about it. Previous versions would just create a unset field. Even -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Geopackages and Datetime fields
Hi Devs / Geopackage masters,
Sorry for this long email, but I have some Geopackage/Datetime questions
of which I hope people here have some knowledge and find some time to
read this...
I'm experimenting with writing time-based model results into
geopackages, and make those results (1 run = 1 geopackage) automatically
available as WMS-t.
-1- First thing I wonder is how gdal/mapserver is handling geopackages
in a multi-threaded environment: if more web=clients are making a lot of
requests, is gdal/mapserver able to open the geopackage in parallel?
Reason I ask is because of the '-shm' and '-wal' files.
-2- Seconds Question is about Datetime handling.
Reading [0] I understand that SQLite (and so gpkg) does not have a
native store format for date/times. And there is something called
'affinity' which determines IF something is stored as integer, float or
string.
I wonder how Mapserver/GDAL is handling filters with datetimes. Trying
to setup mapserver with a geopackage with datetimes as ISO-strings all
worked. But doing the same with a datetime column holding
'unixepoch'-integers was NOT working.
Should this work? Or is filtering only working with ISO datetime strings?
-3- I tried both Mapserver and Geoserver (I know, not gdal...). But
Geoserver is not 'seeing' the datetimes as... well datetimes, but as
strings.
Created an issue for this [1], and wondered how GDAL or other software
was handling this.
In my issue [1] Jukka proposed to use a query like:
SELECT sql FROM sqlite_master
WHERE type = 'table' AND name = 'tablename'
which then gives you the sql to create the table WITH the types in it.
BUT this is not working for 'views' based on that table (my use case...)
Then I found in GDAL sources [2] the following:
PRAGMA table_info('tablename')
which works BOTH for tables AND views !
Question here: would that be the best route to take to make Geotools-lib
more datetime-aware?
Looking in the sqlite descriptions about pragma's [3] it is not very
safe, but ... hey, if GDAL is using it... :-)
Thanks for anybody willing to shed some light in this.
Regards,
Richard Duivenvoorde
[0] https://www.sqlite.org/datatype3.html
[1] https://osgeo-org.atlassian.net/browse/GEOS-9392
[2]
https://github.com/OSGeo/gdal/blob/master/gdal/ogr/ogrsf_frmts/sqlite/ogrsqlitetablelayer.cpp#L525
[3] https://www.sqlite.org/pragma.html
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Speeding up gdalwarp process
I have used gnu parallel with good results. A description can be found here: https://gis.stackexchange.com/questions/217814/full-utilization-of-cpu-cores-with-gdalwarp (https://gis.stackexchange.com/questions/217814/full-utilization-of-cpu-cores-with-gdalwarp) /Lars Schylberg 24 januari 2020 kl. 02:31, "Simon" mailto:twinbi...@protonmail.com?to=%22Simon%22%20)> skrev: Hi gdal-devs, I have a question, if there is some way to use gdalwarp in a clustering system (e.g., Sparks) or with GPU to speed up the process? My task involves re-projection and re-sampling of hundreds of high-resolution images. Any ideas to make use of Sparks or GPU is welcomed. Thank you. Simon ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
