Re: [Qgis-developer] Spatialize attribute columns XY oor WKT, design choice

[Hugo Mercier]

Hi,

I've added a wiki page to discuss the design of this feature here :
http://hub.qgis.org/wiki/quantum-gis/virtual_layers

It's mainly focused on Regis' use case (create a spatial view out of X,Y
coordinates).

Feel free to comment/complete.

Le 19/02/2014 11:46, Nathan Woodrow a écrit :
> On Wed, Feb 19, 2014 at 6:42 PM, Hugo Mercier  > wrote:
> 
> And Spatialite functions handling spatial indexes can be used on
> them ...
> So we could imagine a VirtualQGIS virtual table driver distributed with
> QGIS ...
> Cool ! :)
> 
> 
> Yeah it is pretty cool.  That is where I first got the idea from.
> 
> My main idea was to have a QGIS implementation of that that can sit on
> top of any QGIS layer not just OGR ones.  So instead of having a backend
> of OGRFeature and OGRDatasource it would use QgsDataProvider and QgsFeature.
> 
> It's pretty cool that there is a Python way to do this, will be good to
> prototype something quick then port to C++ for speed.  I might have some
> time tomorrow to try my hand at something basic and if it works we can
> explore it further.
> 
> - Nathan
> 
> 


-- 
Hugo Mercier
Oslandia
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Spatialize attribute columns XY oor WKT, design choice

[Matthias Kuhn]

On Wed 19 Feb 2014 12:50:20 PM CET, Nathan Woodrow wrote:

Well it's just SQLIte so the Spatialite provider should handle it fine
because to it it's just a normal database with tables.

I would also like to expose the data provider API to Python but I
don't think we will need it for this.

- Nathan




Good plan, looking foward.

Matthias
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Spatialize attribute columns XY oor WKT, design choice

[Nathan Woodrow]

Well it's just SQLIte so the Spatialite provider should handle it fine
because to it it's just a normal database with tables.

I would also like to expose the data provider API to Python but I don't
think we will need it for this.

- Nathan


On Wed, Feb 19, 2014 at 9:47 PM, Matthias Kuhn  wrote:

> A bit of a problem I see is that there is still no support for plugin data
> providers. Are you thinking about implementing this as a plugin layer?
>
>
> On Wed 19 Feb 2014 11:46:39 AM CET, Nathan Woodrow wrote:
>
>> On Wed, Feb 19, 2014 at 6:42 PM, Hugo Mercier
>> mailto:hugo.merc...@oslandia.com>> wrote:
>>
>> And Spatialite functions handling spatial indexes can be used on
>> them ...
>> So we could imagine a VirtualQGIS virtual table driver distributed
>> with
>> QGIS ...
>> Cool ! :)
>>
>>
>> Yeah it is pretty cool.  That is where I first got the idea from.
>>
>> My main idea was to have a QGIS implementation of that that can sit on
>> top of any QGIS layer not just OGR ones.  So instead of having a
>> backend of OGRFeature and OGRDatasource it would use QgsDataProvider
>> and QgsFeature.
>>
>> It's pretty cool that there is a Python way to do this, will be good
>> to prototype something quick then port to C++ for speed.  I might have
>> some time tomorrow to try my hand at something basic and if it works
>> we can explore it further.
>>
>> - Nathan
>>
>>
>>
>>
>> ___
>> Qgis-developer mailing list
>> Qgis-developer@lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
>
>
>
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Spatialize attribute columns XY oor WKT, design choice

[Matthias Kuhn]

A bit of a problem I see is that there is still no support for plugin 
data providers. Are you thinking about implementing this as a plugin 
layer?


On Wed 19 Feb 2014 11:46:39 AM CET, Nathan Woodrow wrote:

On Wed, Feb 19, 2014 at 6:42 PM, Hugo Mercier
mailto:hugo.merc...@oslandia.com>> wrote:

And Spatialite functions handling spatial indexes can be used on
them ...
So we could imagine a VirtualQGIS virtual table driver distributed
with
QGIS ...
Cool ! :)


Yeah it is pretty cool.  That is where I first got the idea from.

My main idea was to have a QGIS implementation of that that can sit on
top of any QGIS layer not just OGR ones.  So instead of having a
backend of OGRFeature and OGRDatasource it would use QgsDataProvider
and QgsFeature.

It's pretty cool that there is a Python way to do this, will be good
to prototype something quick then port to C++ for speed.  I might have
some time tomorrow to try my hand at something basic and if it works
we can explore it further.

- Nathan




___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer



___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Spatialize attribute columns XY oor WKT, design choice

[Nathan Woodrow]

On Wed, Feb 19, 2014 at 6:42 PM, Hugo Mercier wrote:

> And Spatialite functions handling spatial indexes can be used on them ...
> So we could imagine a VirtualQGIS virtual table driver distributed with
> QGIS ...
> Cool ! :)
>

Yeah it is pretty cool.  That is where I first got the idea from.

My main idea was to have a QGIS implementation of that that can sit on top
of any QGIS layer not just OGR ones.  So instead of having a backend of
OGRFeature and OGRDatasource it would use QgsDataProvider and QgsFeature.

It's pretty cool that there is a Python way to do this, will be good to
prototype something quick then port to C++ for speed.  I might have some
time tomorrow to try my hand at something basic and if it works we can
explore it further.

- Nathan
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Spatialize attribute columns XY oor WKT, design choice

[Hugo Mercier]

Le 19/02/2014 08:16, Matthias Kuhn a écrit :
> Hi,
> 
> Recently there was a discussion on IRC between NathanW and EvenR
> concerning SQLite's virtual table [1]
> I think this would be a (mighty) generic approach which could also serve
> this use case. I didn't give it much deeper thoughts yet, but I must
> admit it looks promising. Basically you just register a new table with
> SQLite and implement the backend for it. This gives you all the SQLite
> power (where we even have a driver for already) and therefore database
> goodies tailor suited to runtime-generated data.

Hmmm ... sounds very interesting !
Looking for more details, I found this :
https://www.gaia-gis.it/fossil/libspatialite/wiki?name=VirtualOGR
which is a VirtualTable driver for Spatialite giving access to OGR
sources ...
And Spatialite functions handling spatial indexes can be used on them ...
So we could imagine a VirtualQGIS virtual table driver distributed with
QGIS ...
Cool ! :)

> 
> I also had the idea of creating such tables on the fly for iterators
> spanning multiple tables (for JOIN support in queries. Providing a
> provider-independent highlevel query syntax like expressions). If we do
> this I would like to make sure that we have the possibility to get
> access to the parsed query before it is executed. This is necessary to
> be able to optimize the query for databases for which we have our own
> drivers and which support such operations natively (Think of a join of
> two possibly huge postgres tables: you don't want to join them locally).
> I haven't yet verified if this is possible or not, and can therefore not
> say if this approach fulfils the requirements.

Yes. That could be used for joins as well.
I am not sure how SQLite would handle things if you have two virtual
tables and decide to JOIN them in a query ...

> 
> There are python bindings available for virtual tables, so it may be
> possible to implement it this way [2] I think it would be good to gather
> some experience in this direction before implementing it in core. I am
> not even sure how it plays with SpatiaLite.

Thanks for pointing this out. I have to think about this further.
I like it. It also enforces the idea that a Spatialite-based format
(GeoPackage?) should be considered as the native layer format for QGIS.

-- 
Hugo Mercier
Oslandia
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Spatialize attribute columns XY oor WKT, design choice

[Matthias Kuhn]

Hi,

Recently there was a discussion on IRC between NathanW and EvenR 
concerning SQLite's virtual table [1]
I think this would be a (mighty) generic approach which could also 
serve this use case. I didn't give it much deeper thoughts yet, but I 
must admit it looks promising. Basically you just register a new table 
with SQLite and implement the backend for it. This gives you all the 
SQLite power (where we even have a driver for already) and therefore 
database goodies tailor suited to runtime-generated data.


I also had the idea of creating such tables on the fly for iterators 
spanning multiple tables (for JOIN support in queries. Providing a 
provider-independent highlevel query syntax like expressions). If we do 
this I would like to make sure that we have the possibility to get 
access to the parsed query before it is executed. This is necessary to 
be able to optimize the query for databases for which we have our own 
drivers and which support such operations natively (Think of a join of 
two possibly huge postgres tables: you don't want to join them 
locally). I haven't yet verified if this is possible or not, and can 
therefore not say if this approach fulfils the requirements.


There are python bindings available for virtual tables, so it may be 
possible to implement it this way [2] I think it would be good to 
gather some experience in this direction before implementing it in 
core. I am not even sure how it plays with SpatiaLite.


Best
Matthias

[1] https://www.sqlite.org/vtab.html
[2] http://apidoc.apsw.googlecode.com/hg/vtable.html

On Tue 18 Feb 2014 05:59:29 PM CET, Régis Haubourg wrote:

Hi Hugo,
I like the idea!
Sounds clear to me. if core commiters agree, that would be nice to start
such a work for 2.4.
any objection?



--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Spatialize-attribute-columns-XY-oor-WKT-design-choice-tp5104209p5104521.html
Sent from the Quantum GIS - Developer mailing list archive at Nabble.com.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer



___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Spatialize attribute columns XY oor WKT, design choice

[Régis Haubourg]

Hi Hugo, 
I like the idea!
Sounds clear to me. if core commiters agree, that would be nice to start
such a work for 2.4. 
any objection?



--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Spatialize-attribute-columns-XY-oor-WKT-design-choice-tp5104209p5104521.html
Sent from the Quantum GIS - Developer mailing list archive at Nabble.com.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Spatialize attribute columns XY oor WKT, design choice

[Hugo Mercier]

Hi Régis,

Le 17/02/2014 12:08, Régis Haubourg a écrit :
> Hi, 
> Users here ask us a good reliable way to spatialize data from XY or WKT
> columns and I'm wondering what could be the reference architecture to have
> it in core. I am aware of Xytools plugins, delimited text import and
> RTQspider plugin, and also of VRT GDAL format. Questions here are for 2.3
> branch or later. 
> 
> Here are the functionnal requirements:
>  
[snip]
> VRT is IMHO no adaptated since it is not dynamic with csv data, GDAL lacks
> many options to open correctly csv (text delimiters, decimal delimiters...)
> . This is why delimited text exists. It also has no spatial index. 
> It still remains extremly powerfull for UNION, spatial queries thanks to SQL
> dialect...
> 

I think you need something very close to GDAL's VRT, but supported
natively by QGIS.

It could be (just thinking loud) something like a special
QgsVectorProvider that "wraps" another one (or many other ones) with
options like "pointsFromColumns", WKT field, etc. or with the
possibility to have "computed" columns based on QgsExpression
(geometryFromWKT already exists, we may want to add MakePoint, MakeLine,
etc.)
This provider would not allow edition of computed geometries and would
be responsible of indexes and so on.


-- 
Hugo Mercier
Oslandia
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer