date:20210927

Re: [gdal-dev] GDAL installation on Docker not working anymore

2021-09-27 Thread Matheus Cammarosano Hidalgo

The proposed solution applies to GDAL 3.0.4 going forward and I am working
with 2.2.3. It is weird because I haven't changed anything in my
Dockerfile...

Em qui., 23 de set. de 2021 às 13:56, Even Rouault <
even.roua...@spatialys.com> escreveu:

> Perhaps related to https://github.com/OSGeo/gdal/issues/4467 ? where
> upgrading setuptools was suggested
> Le 23/09/2021 à 18:40, Matheus Cammarosano Hidalgo a écrit :
>
> I made a Dockerfile two years ago with a GDAL 2.2.3 install that was
> working fine ultil three weeks ago, when I made a change in my Python code
> (not Dockerfile), but when I tried to build the image I got an
> error installing GDAL.
> I am using a python:3.7 default image from Docker and I have the following
> lines to install GDAL:
> # Install GDAL dependencies
> RUN apt-get install -y python3-pip libgdal-dev locales
>
> # Ensure locales configured correctly
> RUN locale-gen en_US.UTF-8
> ENV LC_ALL='en_US.utf8'
>
> # Set python aliases for python3
> RUN echo 'alias python=python3' >> ~/.bashrc
> RUN echo 'alias pip=pip3' >> ~/.bashrc
>
> # Update C env vars so compiler can find gdal
> ENV CPLUS_INCLUDE_PATH=/usr/include/gdal
> ENV C_INCLUDE_PATH=/usr/include/gdal
>
> RUN pip3 install GDAL==2.2.3
>
> When I try to build the image, I get the following error message:
> ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c
> 'import io, os, sys, setuptools, tokenize; sys.argv[0] =
> '"'"'/tmp/pip-install-9sbmiien/gdal_b1debd2d108d43f38c9919e931126904/setup.py'"'"';
> __file__='"'"'/tmp/pip-install-9sbmiien/gdal_b1debd2d108d43f38c9919e931126904/setup.py'"'"';f
> = getattr(tokenize, '"'"'open'"'"', open)(__file__) if
> os.path.exists(__file__) else io.StringIO('"'"'from setuptools import
> setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"',
> '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))'
> install --record /tmp/pip-record-sp2ip9vm/install-record.txt
> --single-version-externally-managed --compile --install-headers
> /usr/local/include/python3.7m/GDAL Check the logs for full command output.
>
> Does anyone know what is happening and some way to make the installation
> work? Thank you!
>
> Best regards,
> Matheus
>
> ___
> gdal-dev mailing 
> listgdal-dev@lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Does OAPIF paging work as supposed?

2021-09-27 Thread Rahkonen Jukka (MML)

Hi,

At least in this case having INITIAL_REQUEST_PAGE_SIZE=number option would
resolve the practical problem. We have collections with millions of features
and therefore large page size is essential when downloading the whole
collection. On the other hand we have complicated geometries like lake polygons
and reading 1 large geometries for resolving the schema is pretty heavy.
And we have 126 collections in the service that makes 126 x 1 features read
for resolving the schemas if the aim is to clip a small area from all
collections into GeoPackage.

-Jukka Rahkonen-

Lähettäjä: Even Rouault
Lähetetty: maanantai 27. syyskuuta 2021 16.47
Vastaanottaja: Rahkonen Jukka (MML) ;
'gdal-dev@lists.osgeo.org'
Aihe: Re: [gdal-dev] Does OAPIF paging work as supposed?

Jukka,

your analysis is completely correct. Whether this is expected or not probably
depends on situations. Should we have a INITIAL_REQUEST_PAGE_SIZE=number open
option to overload the number of features to retrieve specifically in the first
request... ??

Regarding the spatial filter, it is passed through the OGR API generally after
having queried the schema, and for most OGR datasources it wouldn't influence
the schema, so there isn't much that can be done here, except maybe adding a
BBOX=west,south,east,north open option.

One option to avoid both issues would be for the service to publish DescribedBy
links at the collection level that would point to a XML schema (using a GML
Simple Feature schema profile, such as the one understood by the GML driver) or
a JSON schema (not "too" complicated too). Both are handled by the driver.

Even
Le 27/09/2021 à 15:21, Rahkonen Jukka (MML) a écrit :
Hi,

I tried to read a relatively small BBOX from an OAPIF server but the process
feels rather slow and I do not quite understand what I am seeing in the log.

ogr2ogr -f GPKG test.gpkg
OAPIF:https://avoin-paikkatieto.maanmittauslaitos.fi/maastotiedot/features/v1/?api-key=
-spat 25 65 25.1 65.1 -oo PAGE_SIZE=1 --debug on --config cpl_curl_verbose
yes

Excerpts from the log:

HTTP:
Fetch(https://avoin-paikkatieto.maanmittauslaitos.fi/maastotiedot/features/v1/collections/osoitepiste/items?api-key==json=1)
HTTP: These HTTP headers were set: Accept: application/geo+json,
application/json
...
> GET
> /maastotiedot/features/v1/collections/osoitepiste/items?api-key==json=1
> HTTP/1.1
Host: avoin-paikkatieto.maanmittauslaitos.fi
Accept-Encoding: gzip
Accept: application/geo+json, application/json

* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
...
GeoJSON: First pass: 56.54 %
GeoJSON: First pass: 100.00 %
HTTP:
Fetch(https://avoin-paikkatieto.maanmittauslaitos.fi/maastotiedot/features/v1/collections/osoitepiste/items?api-key==json=1=25,65,25.1014,65.0943)
...
> GET
> /maastotiedot/features/v1/collections/osoitepiste/items?api-key==json=1=25,65,25.1014,65.0943
> HTTP/1.1
...
< Content-Length: 309
GDALVectorTranslate: 0 features written in layer 'osoitepiste'

Do I read right that GDAL is first reading one page, in this time 1
features without BBOX, perhaps for resolving the schema, and then makes a new
query with BBOX? In this case the BBOX query finds nothing. Reading 1
features on the first round and then discarding everything feels too expensive.
Could it be enough to read for example 10 features that is the default page
size on the first round instead of the full page?

-Jukka Rahkonen-

___

gdal-dev mailing list

gdal-dev@lists.osgeo.org

https://lists.osgeo.org/mailman/listinfo/gdal-dev

http://www.spatialys.com

My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Does OAPIF paging work as supposed?

2021-09-27 Thread Even Rouault


Jukka,

your analysis is completely correct. Whether this is expected or not 
probably depends on situations. Should we have a 
INITIAL_REQUEST_PAGE_SIZE=number open option to overload the number of 
features to retrieve specifically in the first request... ??


Regarding the spatial filter, it is passed through the OGR API generally 
after having queried the schema, and for most OGR datasources it 
wouldn't influence the schema, so there isn't much that can be done 
here, except maybe adding a BBOX=west,south,east,north open option.


One option to avoid both issues would be for the service to publish 
DescribedBy links at the collection level that would point to a XML 
schema (using a GML Simple Feature schema profile, such as the one 
understood by the GML driver) or a JSON schema (not "too" complicated 
too). Both are handled by the driver.


Even

Le 27/09/2021 à 15:21, Rahkonen Jukka (MML) a écrit :


Hi,

I tried to read a relatively small BBOX from an OAPIF server but the 
process feels rather slow and I do not quite understand what I am 
seeing in the log.


ogr2ogr -f GPKG test.gpkg 
OAPIF:https://avoin-paikkatieto.maanmittauslaitos.fi/maastotiedot/features/v1/?api-key= 
 
-spat 25 65 25.1 65.1 -oo PAGE_SIZE=1 --debug on --config 
cpl_curl_verbose yes


Excerpts from the log:

HTTP: 
Fetch(https://avoin-paikkatieto.maanmittauslaitos.fi/maastotiedot/features/v1/collections/osoitepiste/items?api-key==json=1 
)


HTTP: These HTTP headers were set: Accept: application/geo+json, 
application/json


…

> GET 
/maastotiedot/features/v1/collections/osoitepiste/items?api-key==json=1
 HTTP/1.1

Host: avoin-paikkatieto.maanmittauslaitos.fi

Accept-Encoding: gzip

Accept: application/geo+json, application/json

* Mark bundle as not supporting multiuse

< HTTP/1.1 200 OK

…

GeoJSON: First pass: 56.54 %

GeoJSON: First pass: 100.00 %

HTTP: 
Fetch(https://avoin-paikkatieto.maanmittauslaitos.fi/maastotiedot/features/v1/collections/osoitepiste/items?api-key==json=1=25,65,25.1014,65.0943 
)


…

> GET 
/maastotiedot/features/v1/collections/osoitepiste/items?api-key==json=1=25,65,25.1014,65.0943
 HTTP/1.1

…

< Content-Length: 309

GDALVectorTranslate: 0 features written in layer 'osoitepiste'

Do I read right that GDAL is first reading one page, in this time 
1 features without BBOX, perhaps for resolving the schema, and 
then makes a new query with BBOX? In this case the BBOX query finds 
nothing. Reading 1 features on the first round and then discarding 
everything feels too expensive. Could it be enough to read for example 
10 features that is the default page size on the first round instead 
of the full page?


-Jukka Rahkonen-


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Does OAPIF paging work as supposed?

2021-09-27 Thread Rahkonen Jukka (MML)

Hi,

I tried to read a relatively small BBOX from an OAPIF server but the process 
feels rather slow and I do not quite understand what I am seeing in the log.

ogr2ogr -f GPKG test.gpkg 
OAPIF:https://avoin-paikkatieto.maanmittauslaitos.fi/maastotiedot/features/v1/?api-key=
 -spat 25 65 25.1 65.1 -oo PAGE_SIZE=1 --debug on --config cpl_curl_verbose 
yes

Excerpts from the log:

HTTP: 
Fetch(https://avoin-paikkatieto.maanmittauslaitos.fi/maastotiedot/features/v1/collections/osoitepiste/items?api-key==json=1)
HTTP: These HTTP headers were set: Accept: application/geo+json, 
application/json
...
> GET 
> /maastotiedot/features/v1/collections/osoitepiste/items?api-key==json=1
>  HTTP/1.1
Host: avoin-paikkatieto.maanmittauslaitos.fi
Accept-Encoding: gzip
Accept: application/geo+json, application/json

* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
...
GeoJSON: First pass: 56.54 %
GeoJSON: First pass: 100.00 %
HTTP: 
Fetch(https://avoin-paikkatieto.maanmittauslaitos.fi/maastotiedot/features/v1/collections/osoitepiste/items?api-key==json=1=25,65,25.1014,65.0943)
...
> GET 
> /maastotiedot/features/v1/collections/osoitepiste/items?api-key==json=1=25,65,25.1014,65.0943
>  HTTP/1.1
...
< Content-Length: 309
GDALVectorTranslate: 0 features written in layer 'osoitepiste'

Do I read right that GDAL is first reading one page, in this time 1 
features without BBOX, perhaps for resolving the schema, and then makes a new 
query with BBOX? In this case the BBOX query finds nothing. Reading 1 
features on the first round and then discarding everything feels too expensive. 
Could it be enough to read for example 10 features that is the default page 
size on the first round instead of the full page?

-Jukka Rahkonen-
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL installation on Docker not working anymore

Re: [gdal-dev] Does OAPIF paging work as supposed?

Re: [gdal-dev] Does OAPIF paging work as supposed?

[gdal-dev] Does OAPIF paging work as supposed?

4 matches

Site Navigation

Mail list logo

Footer information