Re: [QGIS-Developer] Questions wrt. problems with empty layers (SOLVED - somewhat)

[Bo Victor Thomsen]

Hi Richard -

There seems to be some kind of version conflict between the different GIS
software on how to use the OGC definitions for "geometry_columns" - whether
to use the column name "type" og "geometry_type". 

And under any circumstances, QGIS is *not* the guilty part.

IMHO ,The Geoserver community should recognize that different dataproviders
uses different naming conventions for the meta-table  "geometry_columns" and
take the necessary steps to support both versions. 

(Removes brown paperbag over head, looks into mirror and says: Next time,
RTFM ! :-)  

Kind regards

Bo Victor Thomsen
GIS & Database specialist

AestasGIS
Denmark

 

  



--
Sent from: http://osgeo-org.1560.x6.nabble.com/QGIS-Developer-f4099106.html
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Questions wrt. problems with empty layers (SOLVED - somewhat)

[Richard Duivenvoorde]

On 11/05/2019 19.44, Bo Victor Thomsen wrote:
> 
> Retrying the WFS describefeature request on the sql server layers now
> gives the equivalent result as with Postgres:
> 
>   *    ... name="ogr_geometry" nillable="true"
> type="*gml:MultiSurfacePropertyType*"/>...
> 
> 
> There seems to be a discrepancy in how different spatial software 
> interprets how geometry_columns should be structured
> 
>   * GDAL uses "geometry_type" (hence SQLServer uses this too, because it
> was created by ogc2ogc command-line tool in sqlserver) 
>   * PostGIS view use "type"
>   * Spatialite uses "geometry_type"
>   * GeoServer uses "type"
> 
> and so on 
> 
> How do we ever reach "world peace" or solve the climate crisis if we
> even can't agree on the name of friggin' meta table column 

Mmm, well, not by being totalitarian and defining how the view/table
should look like or columns be named :-)

I mean, I think HOW a database defines it's metadata columns is their
internal business.

IF sofware is going to use (OR create) these, that software should
'speak' the different dialects.

So what is you conclusion now?
Is it that Geoserver is requesting the 'type' on databases that use
'geometry_type'?

Or is GDAL creating wrong metadata tables by creating a geometry_type
column which is always called 'geometry_type'.

Whatever is the case I think it is to be fixed there isn't it? QGIS
cannot 'know' what isn't reported by the server back-end?

Regards,

Richard

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Questions wrt. problems with empty layers (SOLVED - somewhat)

[Bo Victor Thomsen]

Hi all -

(for the impatient, read the last part of this mail)

I've been digging a little bit further into my own setup - essentially 
doing the same operations as Richard


I examined the 4 layers i GeoServer

 * groenreg:bykerne: *SqlServer* based layer *with* rows.
   Geometry column defined as "geometry", but defined in
   "geometry_columns" as "MULTIPOLYGON", dimensions = 2 and srid = 25832

 * groenreg:bykerne_pg: *Postgres* based layer*with* rows
   Geometry column defined as "geometry (MULTIPOLYON,25832) and defined
   in "geometry_columns" as "MULTIPOLYGON", dimensions = 2 and srid =
   25832

 * groenreg:bykerne_empty:*Sql-server* based layer *without* rows
   Geometry column defined as "geometry", but defined in
   "geometry_columns" as "MULTIPOLYGON", dimensions = 2 and srid = 25832

 * groenreg:bykerne_pg_empty: *Postgres* based layer *without* rows .
   Geometry column defined as "geometry (MULTIPOLYON,25832) and defined
   in "geometry_columns" as "MULTIPOLYGON", dimensions = 2 and srid =
   25832

The datastore in Geoserver for the layers in SQLServer is set up so the 
location of the geometry_columns is known to GeoServer


The documentation for GeoServer clearly states, that if there is a 
geometry_columns meta table present and is known to GeoServer, GeoServer 
will use information from this table to ascertain the geometry type for 
the layers: 
https://docs.geoserver.org/stable/en/user/data/database/sqlserver.html#using-the-geometry-metadata-table


When I make a WFS describefeature - request for each of the 4 layers I 
get the following result:


 * groenreg:bykerne:  ..

 * groenreg:bykerne_pg:  ... ...

 * groenreg:bykerne_empty:  ..

 * groenreg:bykerne_pg_empty: ... ...

I.e. The layers based on ms-sqlserver (with or without rows) always is 
seen as type="*gml:GeometryPropertyType*" (general geometry, no subtype) 
and layers based on Postgres is seen as  
type="*gml:MultiSurfacePropertyType (*i.e. the correct subtype for the  
specific layer)


My conclusion:
GeoServer is *not* using the information stored in the geometry_columns 
meta table to define the real geometry subtype for a MS-SQLServer based 
layer. Hence QGIS is forced to detect the geometry-type by fetching one 
row and determine the type from this row. If there is no rows, QGIS is 
unable the determine the type.
On the other hand Postgres based layers from GeoServer shows the right 
geometry subtype in the WFS describefeature request. These layers in 
QGIS gets the right type with or without rows in the source database table.


Why GeoServer  isn't using the information in the geometry columns meta 
table is unknown to me. It can be a faulty setup by me or a bug in the 
GeoServer code.


*NB!!! STOP PRESS -**
*
I've just investigated the structure of geometry_columns in my sqlserver 
database and compared it to the geometry_columns structure that 
GeoServer prescribes:
There *is* a bloody difference: The *geometry_type* column in my 
sqlserver geometry_columns table has the "wrong" name. It should be 
named simply *type*


A simple renaming of column *geometry_type* to *type *in table 
geometry_columns solves the issue with the zero-row sqlserver tables as 
layers


(Just turned away from computer and stomping head into wall for about 3 
minutes...



 Back again with a slight headache...)

Retrying the WFS describefeature request on the sql server layers now 
gives the equivalent result as with Postgres:


 *     ..


There seems to be a discrepancy in how different spatial software  
interprets how geometry_columns should be structured


 * GDAL uses "geometry_type" (hence SQLServer uses this too, because it
   was created by ogc2ogc command-line tool in sqlserver)
 * PostGIS view use "type"
 * Spatialite uses "geometry_type"
 * GeoServer uses "type"

and so on 

How do we ever reach "world peace" or solve the climate crisis if we 
even can't agree on the name of friggin' meta table column 


Regards

Bo Victor Thomsen
GIS & Database specialist

AestasGIS
Denmark





Den 10-05-2019 kl. 17:10 skrev Alessandro Pasotti:



On Fri, May 10, 2019 at 5:04 PM Richard Duivenvoorde 
mailto:rdmaili...@duif.net>> wrote:


Hi,

Just tried with a Geoserver (2.15) I have running here.

Moved a shp file with 1 feature in a postgis db, created a wfs service
for it and loaded in QGIS, styled it etc: fine.

Removed the (one) feature from the table, retarted Geoserver, and
had a
look into the requests QGIS is firing [0].

QGIS first does a GetCapabilities request:

/geoserver/wfs?SERVICE=WFS&REQUEST=GetCapabilities&ACCEPTVERSIONS=2.0.0,1.1.0,1.0.0

Then a describefeature request (to determine the type):


geoserver/wfs?SERVICE=WFS&REQUEST=DescribeFeatureType&VERSION=2.0.0&TYPENAME=test:wfspointPostgis

That one responses with



So Geoserver (still?) knows about the geometry type of the table.
Not sure if that is cached? But it is available in

Re: [QGIS-Developer] Questions wrt. problems with empty layers

[Alessandro Pasotti]

On Fri, May 10, 2019 at 5:04 PM Richard Duivenvoorde 
wrote:

> Hi,
>
> Just tried with a Geoserver (2.15) I have running here.
>
> Moved a shp file with 1 feature in a postgis db, created a wfs service
> for it and loaded in QGIS, styled it etc: fine.
>
> Removed the (one) feature from the table, retarted Geoserver, and had a
> look into the requests QGIS is firing [0].
>
> QGIS first does a GetCapabilities request:
>
> /geoserver/wfs?SERVICE=WFS&REQUEST=GetCapabilities&ACCEPTVERSIONS=2.0.0,1.1.0,1.0.0
>
> Then a describefeature request (to determine the type):
>
>
> geoserver/wfs?SERVICE=WFS&REQUEST=DescribeFeatureType&VERSION=2.0.0&TYPENAME=test:wfspointPostgis
>
> That one responses with
>
>  type="gml:PointPropertyType"/>
>
> So Geoserver (still?) knows about the geometry type of the table.
> Not sure if that is cached? But it is available in the
> 'geometry_columns'-view of Postgis (stating type=POINT).
>
> And QGIS keeps the styling, even after several restarts/reloads.
>
> One thing I can think of: is that the postgis table does not have a type
> defined, with someting like:
> geom geometry
> instead of
> geom geometry(Point,28992),
> what I see when the table created via QGIS...
>
> Could it be that
> - geoserver cannot determine the type?
>
> To test I created a minimal table:
> CREATE TABLE public.testgeoms
> (
>   id integer,
>   geom geometry,
>   fid integer NOT NULL DEFAULT nextval('testgeoms_fid_seq'::regclass),
>   CONSTRAINT foo PRIMARY KEY (fid)
> )
>
> At that moment the geometry_columns view shows type=GEOMETRY
> So (I think) without data, geoserver can never determine the type, hence
> shows:
>  type="gml:GeometryPropertyType"/>
> in the DescribeFeatureType response...
>
> And indeed if I then empty the table, and reopen the project the
> WFS-point layer shows an 'img', as it cannot style the zero features.
> AND it does not have type info?
>
> So (confirmed) having that layer with zero (of unknown type) features,
> QGIS forgets the style...
>
> Is that a bug?
>
>
QGIS styles are bound to geometry types, it the type is unknown there is
not much we can do  something similar happens for bad/broken layers.

For that implementation I'm storing the original XML layer style that can
be re-applied in case the "bad layer" is fixed, maybe we can re-use that
mechanism for this use case.




> IF you create the layers with a feature TYPE (geom
> geometry(Point,28992)) then the type will show up in the
> 'geometry_columns'-view and then in the describefeature response.
>
> Without that knowledge QGIS retrieves NO type information (via the
> describefeature) upon loading of the project/layer and (apparently)
> forgets the type in the layer???
>
> Does it work with you if you put the type in the table definitions in
> Postgis (so they show up in the 'geometry_columns'-view)?
> With me it does, but I just did a quick test.
>
> Else I would create an issue for it or let somebody dive into this...
>
> Regards,
>
> Richard Duivenvoorde
>
>
> [0] https://github.com/rduivenvoorde/qgisnetworklogger/tree/model_rewrite
>
>
> On 09/05/2019 22.22, Bo Victor Thomsen wrote:
> > Hi Lars and other members of this list..
> >
> > I feel somewhat obligated to help Lars since it's a problem he has
> > inherited from me. (Sorry about that...)
> >
> > To sum it up:
> >
> > If QGIS connects to a WFS based layer from Geoserver using an existing
> > project with styled layers and one or more layers containing /zero/
> > objects, QGIS can't determine the object type (point, line-string,
> > polygon...) for those layers. QGIS will not show legends  and promptly
> > forget any layer information stored in the project related to the
> > object-type, ex. styling information.
> >
> > And what is worse - this is a permanent situation: If you save the
> > project in the above-mentioned state and reopen it at a later time -
> > where the layers /now/ contains objects - QGIS has forgotten the styling
> > and other information, so the layers stays invisible and without the
> > former styling.
> >
> > (long posting...)
> >
> > I've done some tests to determine the root cause of this problem:
> >
> >   * Setup of a Postgres database /and /a SQL Server database.
> >
> >   * Loaded the same MapInfo-dataset twice into each database, called
> > them tableA and tableB, and populated the meta-table
> > "dbo.geometry_columns" with the correct information for tableA and
> > tableB in SQLServer.
> >
> >   * Installed a fresh Geoserver 2.15 and MapServer TinyOWS WFS-T server
> >
> >   * Setup a Workspace in Geoserver with 4 layers: tableA and -B from
> > SQLServer and the same tables from Postgres using WFS.
> >
> >   * In Geoserver, the datastore definition for the SQLServer tables has
> > a pointer to the "dbo.geometry_columns" meta-table.
> >
> >   * Setup of TinyOWS with tableA and -B from Postgres
> >
> >   * And lastly: Made a project in QGIS 3.4.7 /Windows 64bit with the 6
> > WFS layers:
> >
> >   o tableA

Re: [QGIS-Developer] Questions wrt. problems with empty layers

[Richard Duivenvoorde]

Hi,

Just tried with a Geoserver (2.15) I have running here.

Moved a shp file with 1 feature in a postgis db, created a wfs service
for it and loaded in QGIS, styled it etc: fine.

Removed the (one) feature from the table, retarted Geoserver, and had a
look into the requests QGIS is firing [0].

QGIS first does a GetCapabilities request:
/geoserver/wfs?SERVICE=WFS&REQUEST=GetCapabilities&ACCEPTVERSIONS=2.0.0,1.1.0,1.0.0

Then a describefeature request (to determine the type):

geoserver/wfs?SERVICE=WFS&REQUEST=DescribeFeatureType&VERSION=2.0.0&TYPENAME=test:wfspointPostgis

That one responses with



So Geoserver (still?) knows about the geometry type of the table.
Not sure if that is cached? But it is available in the
'geometry_columns'-view of Postgis (stating type=POINT).

And QGIS keeps the styling, even after several restarts/reloads.

One thing I can think of: is that the postgis table does not have a type
defined, with someting like:
geom geometry
instead of
geom geometry(Point,28992),
what I see when the table created via QGIS...

Could it be that
- geoserver cannot determine the type?

To test I created a minimal table:
CREATE TABLE public.testgeoms
(
  id integer,
  geom geometry,
  fid integer NOT NULL DEFAULT nextval('testgeoms_fid_seq'::regclass),
  CONSTRAINT foo PRIMARY KEY (fid)
)

At that moment the geometry_columns view shows type=GEOMETRY
So (I think) without data, geoserver can never determine the type, hence
shows:

in the DescribeFeatureType response...

And indeed if I then empty the table, and reopen the project the
WFS-point layer shows an 'img', as it cannot style the zero features.
AND it does not have type info?

So (confirmed) having that layer with zero (of unknown type) features,
QGIS forgets the style...

Is that a bug?

IF you create the layers with a feature TYPE (geom
geometry(Point,28992)) then the type will show up in the
'geometry_columns'-view and then in the describefeature response.

Without that knowledge QGIS retrieves NO type information (via the
describefeature) upon loading of the project/layer and (apparently)
forgets the type in the layer???

Does it work with you if you put the type in the table definitions in
Postgis (so they show up in the 'geometry_columns'-view)?
With me it does, but I just did a quick test.

Else I would create an issue for it or let somebody dive into this...

Regards,

Richard Duivenvoorde


[0] https://github.com/rduivenvoorde/qgisnetworklogger/tree/model_rewrite


On 09/05/2019 22.22, Bo Victor Thomsen wrote:
> Hi Lars and other members of this list..
> 
> I feel somewhat obligated to help Lars since it's a problem he has
> inherited from me. (Sorry about that...) 
> 
> To sum it up:
> 
> If QGIS connects to a WFS based layer from Geoserver using an existing
> project with styled layers and one or more layers containing /zero/
> objects, QGIS can't determine the object type (point, line-string,
> polygon...) for those layers. QGIS will not show legends  and promptly
> forget any layer information stored in the project related to the
> object-type, ex. styling information.
> 
> And what is worse - this is a permanent situation: If you save the
> project in the above-mentioned state and reopen it at a later time -
> where the layers /now/ contains objects - QGIS has forgotten the styling
> and other information, so the layers stays invisible and without the
> former styling.
> 
> (long posting...)
> 
> I've done some tests to determine the root cause of this problem:
> 
>   * Setup of a Postgres database /and /a SQL Server database.
> 
>   * Loaded the same MapInfo-dataset twice into each database, called
> them tableA and tableB, and populated the meta-table
> "dbo.geometry_columns" with the correct information for tableA and
> tableB in SQLServer.
> 
>   * Installed a fresh Geoserver 2.15 and MapServer TinyOWS WFS-T server
> 
>   * Setup a Workspace in Geoserver with 4 layers: tableA and -B from
> SQLServer and the same tables from Postgres using WFS.
> 
>   * In Geoserver, the datastore definition for the SQLServer tables has
> a pointer to the "dbo.geometry_columns" meta-table.
> 
>   * Setup of TinyOWS with tableA and -B from Postgres
> 
>   * And lastly: Made a project in QGIS 3.4.7 /Windows 64bit with the 6
> WFS layers:
> 
>   o tableA is shown 3 times:
> 
>  1. From SQLServer through GeoServer;
>  2. From Postgres through GeoServer
>  3. From Postgres through Tiny-WFS
> 
>   o tableB is shown 3 times using the same setup as tableA
> 
> All 6 layers in QGIS was shown correctly and the object type for all
> layers was recognized.
> 
> After this I /deleted/ all rows from tableB in /both/ MS-SQLServer and
> in Postgres and reopened the same QGIS project:
> 
>   * The layer showing (the zero objects) tableB from SQLServer through
> GeoServer was not in order.
>   * The 5 other layers was shown correctly. Legends was correct and
> styling was not deleted
> 
> I'm dra

Re: [QGIS-Developer] Questions wrt. problems with empty layers

[Bo Victor Thomsen]

Hi Lars and other members of this list..

I feel somewhat obligated to help Lars since it's a problem he has 
inherited from me. (Sorry about that...)


To sum it up:

If QGIS connects to a WFS based layer from Geoserver using an existing 
project with styled layers and one or more layers containing /zero/ 
objects, QGIS can't determine the object type (point, line-string, 
polygon...) for those layers. QGIS will not show legends  and promptly 
forget any layer information stored in the project related to the 
object-type, ex. styling information.


And what is worse - this is a permanent situation: If you save the 
project in the above-mentioned state and reopen it at a later time - 
where the layers /now/ contains objects - QGIS has forgotten the styling 
and other information, so the layers stays invisible and without the 
former styling.


(long posting...)

I've done some tests to determine the root cause of this problem:

 * Setup of a Postgres database /and /a SQL Server database.

 * Loaded the same MapInfo-dataset twice into each database, called
   them tableA and tableB, and populated the meta-table
   "dbo.geometry_columns" with the correct information for tableA and
   tableB in SQLServer.

 * Installed a fresh Geoserver 2.15 and MapServer TinyOWS WFS-T server

 * Setup a Workspace in Geoserver with 4 layers: tableA and -B from
   SQLServer and the same tables from Postgres using WFS.

 * In Geoserver, the datastore definition for the SQLServer tables has
   a pointer to the "dbo.geometry_columns" meta-table.

 * Setup of TinyOWS with tableA and -B from Postgres

 * And lastly: Made a project in QGIS 3.4.7 /Windows 64bit with the 6
   WFS layers:

 o tableA is shown 3 times:

1.  From SQLServer through GeoServer;
2.  From Postgres through GeoServer
3.  From Postgres through Tiny-WFS

 o tableB is shown 3 times using the same setup as tableA

All 6 layers in QGIS was shown correctly and the object type for all 
layers was recognized.


After this I /deleted/ all rows from tableB in /both/ MS-SQLServer and 
in Postgres and reopened the same QGIS project:


 * The layer showing (the zero objects) tableB from SQLServer through
   GeoServer was not in order.
 * The 5 other layers was shown correctly. Legends was correct and
   styling was not deleted

I'm drawing the following conclusion:

 * GeoServer WFS connections (maybe in conjunction with QGIS) doesn't
   work properly when the dataprovider is MS-SQLServer and the layers
   contains zero objects. More precisely: GeoServer's SQLServer
   dataprovider can't or don't use geometry_columns meta-table defined
   in the datastore as described in:
   
https://docs.geoserver.org/stable/en/user/data/database/sqlserver.html#using-the-geometry-metadata-table
   .

After this test I've tried to change the placement and existence of the 
"geometry_columns" metadata table:


 * No value
 * dbo.geometry_column (ordinary placement)
 * data.geometry_column (I made a copy of the geometry_columns table
   and placed it in the same schema as the test tables)
 * geometry_columns (without schema definition)

Nothing of the above worked.

@Lars:
You might change your database system to Postgres and perhaps replace 
GeoServer with TinyOWS for WFS purposes (ducks and and run) .
Or getting the issue investigated and fixed using some GeoServer core 
developer, perhaps in cooperation with a QGIS core developer?
Apparently the problem is fixable, since neither TinyOWS or 
GeoServer/Postgres have the same  issues.


Obviously, the above is a far cry from a really rigorous investigation 
of the problem, but it points at some of the "pain" points


Regards

Bo Victor Thomsen

AestasGIS
Denmark

Den 09-05-2019 kl. 13:36 skrev Lars I. Nielsen, LIFA A/S:


Hi developers.

I asked this question on the user’s list, so far without any answers, 
but have been advised to ask here as well.


I’m working with a number of projects, which statically open a fixed 
number of layers. All layers are fetched using WFS / WFS-T.


Each layer has an extensive rule based styling, and an extensive 
field/attribute dialog setup. This is saved in the project files.


Our plugin changes the data provider filter for many layers, when the 
user works with the data. These filter expressions are also saved in 
the project, of course.


Our problem is, that if the user accidently chooses a work area (= new 
data provider filter), which contain no data in any layer, those 
layers lose all styling etc. And it’s apparently impossible to recover 
these styles etc. by changing the filter back.


So basically the user’s project is unrecoverably trashed at this point 
in time.


A - Is this a correct description of how QGIS handles “no-data” layers ?

B - Are there any standard ways to handle this in QGIS (using metadata 
or otherwise) in an automated fashion (i.e. not manually) ?


C - If the answers to A and B are yes and no resp., are there any 
plans to remedy this unnec

[QGIS-Developer] Questions wrt. problems with empty layers

[Lars I. Nielsen, LIFA A/S]

Hi developers.

I asked this question on the user's list, so far without any answers, but have 
been advised to ask here as well.


I'm working with a number of projects, which statically open a fixed number of 
layers. All layers are fetched using WFS / WFS-T.

Each layer has an extensive rule based styling, and an extensive 
field/attribute dialog setup. This is saved in the project files.

Our plugin changes the data provider filter for many layers, when the user 
works with the data. These filter expressions are also saved in the project, of 
course.

Our problem is, that if the user accidently chooses a work area (= new data 
provider filter), which contain no data in any layer, those layers lose all 
styling etc. And it's apparently impossible to recover these styles etc. by 
changing the filter back.

So basically the user's project is unrecoverably trashed at this point in time.

A - Is this a correct description of how QGIS handles "no-data" layers ?

B - Are there any standard ways to handle this in QGIS (using metadata or 
otherwise) in an automated fashion (i.e. not manually) ?

C - If the answers to A and B are yes and no resp., are there any plans to 
remedy this unnecessarily serious problem ?


Might it not be solved simply by adding an (optional?) layer metadata key, that 
stores the last used topology, and revert to this if no data are found on the 
layer ?


Using 3.4.5 x64 (Danish)


Med venlig hilsen
Lars I. Nielsen, LIFA A/S
GIS-konsulent, FME Certified Professional

[Beskrivelse: http://website.lifa.dk/lsp.gif]
T

6313 6800

@

l...@lifa.dk

D

6313 6849

W

www.lifa.dk

M

2492 4866

CVR

20937289


[Beskrivelse: 
C:\Users\lin\AppData\Roaming\Microsoft\signatures\21x21-Images-Get-L749-l8.png]
Følg os på LinkedIn og læs de seneste nyheder fra LIFA A/S
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer