Re: [QGIS-Developer] Understanding plugin management in QGIS - exposing QgsPluginRegistry to Python?

[Nyall Dawson]

On Sun, 15 Mar 2020 at 09:18, Matthias Kuhn  wrote:
>
> Hi Sebastian, Borys,
>
> What about the opposite way: moving things to C++ and leaving only a
> very thing wrapper for interacting with the plugins in place?

That would be my preference too -- the overall movement in QGIS code
is to reduce the overall amount of Python code in master, not the
reverse.

> The rational is that plugin management is not performance-critical and could 
> become much more easy to maintain,

Actually, over time we've found that the opposite is true -- the
Python code in QGIS master is the **most** work to maintain, and a
constant source of regressions. That's due to the nature of the Python
language itself, not a reflection of the quality of the code...

Nyall


>
> That should get rid of duplicated code and be compatible with everything.
>
> Best regards
>
> Matthias
>
> On 3/14/20 10:52 PM, Borys Jurgiel wrote:
> > Hi Sebastian,
> >
> > The reason the whole infrastructure is split between Plugin Manager and 
> > Python
> > Plugin Installer (and Python utils) is Python support is optional in QGIS.
> >
> > If QGIS is compiled without Python Support, the Plugin Manager still 
> > displays
> > all C++ plugins, it just lacks Python ones (and of course install/deinstall
> > funcions).
> >
> > I admit the the infrastructure became a bit complicated ;) However, porting
> > the Plugin Manager to Python would require making Python a hard dependency 
> > of
> > QGIS GUI (or we should drop C++ plugins by porting them all to core, what is
> > discussed from time to time and doesn't seem to may happen).
> >
> > Regards,
> >
> > Borys
> >
> >
> > Dnia sobota, 14 marca 2020 21:10:24 CET Sebastian M. Ernst pisze:
> >> Hi folks,
> >>
> >> out of curiosity I have started poking around in QGIS' code related to
> >> plugin management. It's an interesting mix of C++ and Python code. Even
> >> the plugin management GUI itself is partially C++ (the main window,
> >> QgsPluginManager[Interface], and part of its logic) and partially Python
> >> (all further dialogs and their logic). As an experiment for now, I am
> >> looking into how I could port at least the entire GUI to Python and
> >> disentangle some of the knots. The rational is that plugin management is
> >> not performance-critical and could become much more easy to maintain,
> >> among a lot of other interesting benefits that a potentially fully
> >> Python-based solution could offer. But that's a different story. For
> >> now, I think I have hit a roadblock and was wondering whether anyone
> >> could enlighten me. I'd greatly appreciate any help.
> >>
> >> I *believe* I have a fairly good understanding of what happens
> >> underneath `/python/pyplugin_installer/`. The class `QgsPluginInstaller`
> >> from `installer.py` appears to be a Python API that is called from the
> >> C++-interface (`QgsPluginManagerInterface`) through `QgsPythonRunner`.
> >> Most of the C++ appears to be located underneath
> >> `/src/app/pluginmanager/` (with a single ui-file elsewhere,
> >> `/src/ui/qgspluginmanagerbase.ui`). Having analyzed the C++ class
> >> `QgsPluginManager` I believe most of the interface can easily ported to
> >> Python.
> >>
> >> The tricky bit appears to be `/src/app/qgspluginregistry.cpp`. It offers
> >> a class named `QgsPluginRegistry` (which is being used by
> >> `QgsPluginManager`). I have not found a way to access this class (or
> >> an/the instance of it) from Python. I figure it is required to handle
> >> both Python *and* C++ plugins. `QgsPluginRegistry` makes heavy use of
> >> `mPythonUtils`, which appears to be a C++ wrapper around
> >> `/python/utils.py` (through `/src/python/qgspythonutilsimpl.cpp`). So
> >> this portion of `QgsPluginRegistry` (about 50%) can be rewritten in
> >> Python rather quickly. Methods such as `loadCppPlugin` or
> >> `unloadCppPlugin` are a lot more problematic. As far as I can tell,
> >> those would need to remain in C++ and they would need to be exposed to
> >> Python somehow (if they are not already).
> >>
> >> Have I found all relevant components of QGIS responsible for plugin
> >> management? Does my analysis contain any major flaws? Is there a way to
> >> manage C++ plugins (i.e. to call `loadCppPlugin` and `unloadCppPlugin`)
> >> from Python?
> >>
> >> Best regards,
> >> Sebastian
> >> ___
> >> QGIS-Developer mailing list
> >> QGIS-Developer@lists.osgeo.org
> >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >
> >
> >
> > ___
> > QGIS-Developer mailing list
> > QGIS-Developer@lists.osgeo.org
> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: 

Re: [QGIS-Developer] Understanding plugin management in QGIS - exposing QgsPluginRegistry to Python?

[Matthias Kuhn]

Hi Sebastian, Borys,

What about the opposite way: moving things to C++ and leaving only a 
very thing wrapper for interacting with the plugins in place?


That should get rid of duplicated code and be compatible with everything.

Best regards

Matthias

On 3/14/20 10:52 PM, Borys Jurgiel wrote:

Hi Sebastian,

The reason the whole infrastructure is split between Plugin Manager and Python
Plugin Installer (and Python utils) is Python support is optional in QGIS.

If QGIS is compiled without Python Support, the Plugin Manager still displays
all C++ plugins, it just lacks Python ones (and of course install/deinstall
funcions).

I admit the the infrastructure became a bit complicated ;) However, porting
the Plugin Manager to Python would require making Python a hard dependency of
QGIS GUI (or we should drop C++ plugins by porting them all to core, what is
discussed from time to time and doesn't seem to may happen).

Regards,

Borys


Dnia sobota, 14 marca 2020 21:10:24 CET Sebastian M. Ernst pisze:

Hi folks,

out of curiosity I have started poking around in QGIS' code related to
plugin management. It's an interesting mix of C++ and Python code. Even
the plugin management GUI itself is partially C++ (the main window,
QgsPluginManager[Interface], and part of its logic) and partially Python
(all further dialogs and their logic). As an experiment for now, I am
looking into how I could port at least the entire GUI to Python and
disentangle some of the knots. The rational is that plugin management is
not performance-critical and could become much more easy to maintain,
among a lot of other interesting benefits that a potentially fully
Python-based solution could offer. But that's a different story. For
now, I think I have hit a roadblock and was wondering whether anyone
could enlighten me. I'd greatly appreciate any help.

I *believe* I have a fairly good understanding of what happens
underneath `/python/pyplugin_installer/`. The class `QgsPluginInstaller`
from `installer.py` appears to be a Python API that is called from the
C++-interface (`QgsPluginManagerInterface`) through `QgsPythonRunner`.
Most of the C++ appears to be located underneath
`/src/app/pluginmanager/` (with a single ui-file elsewhere,
`/src/ui/qgspluginmanagerbase.ui`). Having analyzed the C++ class
`QgsPluginManager` I believe most of the interface can easily ported to
Python.

The tricky bit appears to be `/src/app/qgspluginregistry.cpp`. It offers
a class named `QgsPluginRegistry` (which is being used by
`QgsPluginManager`). I have not found a way to access this class (or
an/the instance of it) from Python. I figure it is required to handle
both Python *and* C++ plugins. `QgsPluginRegistry` makes heavy use of
`mPythonUtils`, which appears to be a C++ wrapper around
`/python/utils.py` (through `/src/python/qgspythonutilsimpl.cpp`). So
this portion of `QgsPluginRegistry` (about 50%) can be rewritten in
Python rather quickly. Methods such as `loadCppPlugin` or
`unloadCppPlugin` are a lot more problematic. As far as I can tell,
those would need to remain in C++ and they would need to be exposed to
Python somehow (if they are not already).

Have I found all relevant components of QGIS responsible for plugin
management? Does my analysis contain any major flaws? Is there a way to
manage C++ plugins (i.e. to call `loadCppPlugin` and `unloadCppPlugin`)
from Python?

Best regards,
Sebastian
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer




___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer


___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Understanding plugin management in QGIS - exposing QgsPluginRegistry to Python?

[Borys Jurgiel]

Hi Sebastian,

The reason the whole infrastructure is split between Plugin Manager and Python 
Plugin Installer (and Python utils) is Python support is optional in QGIS. 

If QGIS is compiled without Python Support, the Plugin Manager still displays 
all C++ plugins, it just lacks Python ones (and of course install/deinstall 
funcions).

I admit the the infrastructure became a bit complicated ;) However, porting 
the Plugin Manager to Python would require making Python a hard dependency of 
QGIS GUI (or we should drop C++ plugins by porting them all to core, what is 
discussed from time to time and doesn't seem to may happen).

Regards,

Borys


Dnia sobota, 14 marca 2020 21:10:24 CET Sebastian M. Ernst pisze:
> Hi folks,
> 
> out of curiosity I have started poking around in QGIS' code related to
> plugin management. It's an interesting mix of C++ and Python code. Even
> the plugin management GUI itself is partially C++ (the main window,
> QgsPluginManager[Interface], and part of its logic) and partially Python
> (all further dialogs and their logic). As an experiment for now, I am
> looking into how I could port at least the entire GUI to Python and
> disentangle some of the knots. The rational is that plugin management is
> not performance-critical and could become much more easy to maintain,
> among a lot of other interesting benefits that a potentially fully
> Python-based solution could offer. But that's a different story. For
> now, I think I have hit a roadblock and was wondering whether anyone
> could enlighten me. I'd greatly appreciate any help.
> 
> I *believe* I have a fairly good understanding of what happens
> underneath `/python/pyplugin_installer/`. The class `QgsPluginInstaller`
> from `installer.py` appears to be a Python API that is called from the
> C++-interface (`QgsPluginManagerInterface`) through `QgsPythonRunner`.
> Most of the C++ appears to be located underneath
> `/src/app/pluginmanager/` (with a single ui-file elsewhere,
> `/src/ui/qgspluginmanagerbase.ui`). Having analyzed the C++ class
> `QgsPluginManager` I believe most of the interface can easily ported to
> Python.
> 
> The tricky bit appears to be `/src/app/qgspluginregistry.cpp`. It offers
> a class named `QgsPluginRegistry` (which is being used by
> `QgsPluginManager`). I have not found a way to access this class (or
> an/the instance of it) from Python. I figure it is required to handle
> both Python *and* C++ plugins. `QgsPluginRegistry` makes heavy use of
> `mPythonUtils`, which appears to be a C++ wrapper around
> `/python/utils.py` (through `/src/python/qgspythonutilsimpl.cpp`). So
> this portion of `QgsPluginRegistry` (about 50%) can be rewritten in
> Python rather quickly. Methods such as `loadCppPlugin` or
> `unloadCppPlugin` are a lot more problematic. As far as I can tell,
> those would need to remain in C++ and they would need to be exposed to
> Python somehow (if they are not already).
> 
> Have I found all relevant components of QGIS responsible for plugin
> management? Does my analysis contain any major flaws? Is there a way to
> manage C++ plugins (i.e. to call `loadCppPlugin` and `unloadCppPlugin`)
> from Python?
> 
> Best regards,
> Sebastian
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer




___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[QGIS-Developer] Understanding plugin management in QGIS - exposing QgsPluginRegistry to Python?

[Sebastian M. Ernst]

Hi folks,

out of curiosity I have started poking around in QGIS' code related to
plugin management. It's an interesting mix of C++ and Python code. Even
the plugin management GUI itself is partially C++ (the main window,
QgsPluginManager[Interface], and part of its logic) and partially Python
(all further dialogs and their logic). As an experiment for now, I am
looking into how I could port at least the entire GUI to Python and
disentangle some of the knots. The rational is that plugin management is
not performance-critical and could become much more easy to maintain,
among a lot of other interesting benefits that a potentially fully
Python-based solution could offer. But that's a different story. For
now, I think I have hit a roadblock and was wondering whether anyone
could enlighten me. I'd greatly appreciate any help.

I *believe* I have a fairly good understanding of what happens
underneath `/python/pyplugin_installer/`. The class `QgsPluginInstaller`
from `installer.py` appears to be a Python API that is called from the
C++-interface (`QgsPluginManagerInterface`) through `QgsPythonRunner`.
Most of the C++ appears to be located underneath
`/src/app/pluginmanager/` (with a single ui-file elsewhere,
`/src/ui/qgspluginmanagerbase.ui`). Having analyzed the C++ class
`QgsPluginManager` I believe most of the interface can easily ported to
Python.

The tricky bit appears to be `/src/app/qgspluginregistry.cpp`. It offers
a class named `QgsPluginRegistry` (which is being used by
`QgsPluginManager`). I have not found a way to access this class (or
an/the instance of it) from Python. I figure it is required to handle
both Python *and* C++ plugins. `QgsPluginRegistry` makes heavy use of
`mPythonUtils`, which appears to be a C++ wrapper around
`/python/utils.py` (through `/src/python/qgspythonutilsimpl.cpp`). So
this portion of `QgsPluginRegistry` (about 50%) can be rewritten in
Python rather quickly. Methods such as `loadCppPlugin` or
`unloadCppPlugin` are a lot more problematic. As far as I can tell,
those would need to remain in C++ and they would need to be exposed to
Python somehow (if they are not already).

Have I found all relevant components of QGIS responsible for plugin
management? Does my analysis contain any major flaws? Is there a way to
manage C++ plugins (i.e. to call `loadCppPlugin` and `unloadCppPlugin`)
from Python?

Best regards,
Sebastian
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Doc Travis

[Alessandro Pasotti]

Hi,

you don't need a Travis sorcerer, the commands do not even run on a
local installation.

Try this patch: https://termbin.com/mmb7



On Sat, Mar 14, 2020 at 5:59 PM matteo  wrote:
>
> Hi all,
>
> we (Richard, me, Harrissou) are trying to build the QGI doc (3.19) with
> the rtd theme (thank to Harrissou and Richard jobs).
>
> All the progress are on branch qgisdoc2020 in the official
> QGIS-Documentation repository:
>
> https://github.com/qgis/QGIS-Documentation/tree/qgisdoc2020
>
> But we are stuck with Travis:
>
> https://travis-ci.org/github/qgis/QGIS-Documentation/builds/662445827?utm_source=github_status_medium=notification
>
> can some Travis guru have a look and help us fix it?
>
> Cheers and thanks
>
> Matteo
>
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer



-- 
Alessandro Pasotti
w3:   www.itopen.it
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Doc Travis

[Denis Rouzaud]

>
>
>
>
>
> -- Forwarded message --
> From: Denis Rouzaud 
> To: matteo 
> Cc: qgis-developer , qgis-community <
> qgis-community-t...@lists.osgeo.org>, Richard Duivenvoorde <
> rich...@duif.net>
> Bcc:
> Date: Sat, 14 Mar 2020 18:06:31 +0100
> Subject: Re: [QGIS-Developer] QGIS Doc Travis
>
>
> Le sam. 14 mars 2020 à 17:59, matteo  a écrit :
>
>> Hi all,
>>
>> we (Richard, me, Harrissou) are trying to build the QGI doc (3.19) with
>> the rtd theme (thank to Harrissou and Richard jobs).
>>
>> All the progress are on branch qgisdoc2020 in the official
>> QGIS-Documentation repository:
>>
>> https://github.com/qgis/QGIS-Documentation/tree/qgisdoc2020
>>
>> But we are stuck with Travis:
>>
>>
>> https://travis-ci.org/github/qgis/QGIS-Documentation/builds/662445827?utm_source=github_status_medium=notification
>>
>> can some Travis guru have a look and help us fix it?
>
>
Apparently you're not looking for a Travis guru, but a Sphinx guru:
sphinx-build: error: the following arguments are required: outputdir,
filenames
https://travis-ci.org/github/qgis/QGIS-Documentation/builds/662445827#L750

I would recommend you to run the same on your machine, run interactively
the Docker container so you can try what's going wrong with the
sphinx-build command.

>
>
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Doc Travis

[matteo]

sorry, 3.10 :)
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[QGIS-Developer] QGIS Doc Travis

[matteo]

Hi all,

we (Richard, me, Harrissou) are trying to build the QGI doc (3.19) with
the rtd theme (thank to Harrissou and Richard jobs).

All the progress are on branch qgisdoc2020 in the official
QGIS-Documentation repository:

https://github.com/qgis/QGIS-Documentation/tree/qgisdoc2020

But we are stuck with Travis:

https://travis-ci.org/github/qgis/QGIS-Documentation/builds/662445827?utm_source=github_status_medium=notification

can some Travis guru have a look and help us fix it?

Cheers and thanks

Matteo

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS "fast" mode

[Tomas Straupis]

2020-03-14, št, 13:58 Régis Haubourg rašė:
> Check the project property option named "trust". This is exactly what it is 
> supposed to do.
> I admit the wording is not nice, any better idea is welcome.

  You mean this one: "Trust project when data source has no metadata"?
  I read the wording that it tries to check metadata and if that is
unsuccessful - trusts project data. My idea is to not even try
re-checking in metadata or anywhere else - always use data from
project and only execute queries for actual job to be done (also do
not try to filter on geometry type etc).

  Practical example: even simple things like re-checking table columns
in metadata (oracle data dictionary) could take 10-15 seconds when we
have ~100 layers. And on very high load these 10-15 seconds are
unwanted (and they will never find anything different than data in
project file).

> @Tomas: are you sure it is doing full table scans? Most database have
> some 'metadata' or cached metadata tables which could be used for that.

  For geometry checking in our scenario it is doing very badly because
data comes from a view which is optimised to be used with spatial
filter (spatial queries take ~100ms), but when we try to query
distinct geometry types (even from random 100 rows) usage schema is
different and it takes ages. Haven't checked but I'm pretty sure it
involves full scans because it takes hours and we do not want to tune
the view with hints for geometry type checking as that will decrease
performance of actually useful queries fetching features.
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS "fast" mode

[Richard Duivenvoorde]

Hi,

Related: having projects with several WMS or WFS server layers also are
opening very slowly (even when NOT visible). It would be nice to have
'lazy' loading, so only load layer which are visible?

@Tomas: are you sure it is doing full table scans? Most database have
some 'metadata' or cached metadata tables which could be used for that.

Regards,

Richard Duivenvoorde

On 3/14/20 12:55 PM, Tomas Straupis wrote:
> Hello
> 
>   I would like to know opinion of QGIS developer community on the
> question of "self-fixing queries" in the context of database layers.
> 
>   When you add a new database layer, QGIS queries required data, asks
> user for other information and then saves it in the project file.
>   Now when you re-open the project, QGIS is re-querying a lot of data
> which is already saved in the project: geometry type, srid, attributes
> (table columns/types).
>   The downsides are:
>   * these "re-checking" queries take considerable time on large
> databases: with millions of records from 1minute up to several hours,
> database is tuned - WMS/WFS queries finish in ~100ms.
>   * in QGIS server environment these re-checking queries are executed
> each time when apache launches a new QGIS server process
>   * QGIS might by itself (without human interaction) decide to  work
> differently compared to what it was asked to do when the project was
> created if it finds that "something has changed"
>   * re-checking during project opening is not 100% correct, as if
> changes are possible in principle, then data/schema could change
> during the time QGIS (Desktop/Server) is open/running
>   * in environments other than development (testing, qa, production),
> database schema should never change without some clear procedure which
> should also include updating QGIS project files and testing them as
> required, therefore such re-checkings should never be required on
> testing/production environments.
> 
>   I do understand that the current situation/understanding is
> different from mine, so in order to satisfy everybody, maybe it would
> be feasible to add say an environment variable like QGIS_FAST=TRUE (or
> QGIS_PRODUCTION=TRUE) and if this is set - re-checking queries could
> be skipped? We've been running modified QGIS without re-checking in
> production for months and are very happy with results. This should
> also improve the benchmarks of QGIS Server as it is known that initial
> opening of the project takes considerable amount of time.
> 
>   What is your opinion?
> 
>   Thank you
> 

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS "fast" mode

[Régis Haubourg]

Hi Tomas.
Check the project property option named "trust". This is exactly what it is
supposed to do.
I admit the wording is not nice, any better idea is welcome.
Best regards


Le sam. 14 mars 2020 à 12:55, Tomas Straupis  a
écrit :

> Hello
>
>   I would like to know opinion of QGIS developer community on the
> question of "self-fixing queries" in the context of database layers.
>
>   When you add a new database layer, QGIS queries required data, asks
> user for other information and then saves it in the project file.
>   Now when you re-open the project, QGIS is re-querying a lot of data
> which is already saved in the project: geometry type, srid, attributes
> (table columns/types).
>   The downsides are:
>   * these "re-checking" queries take considerable time on large
> databases: with millions of records from 1minute up to several hours,
> database is tuned - WMS/WFS queries finish in ~100ms.
>   * in QGIS server environment these re-checking queries are executed
> each time when apache launches a new QGIS server process
>   * QGIS might by itself (without human interaction) decide to  work
> differently compared to what it was asked to do when the project was
> created if it finds that "something has changed"
>   * re-checking during project opening is not 100% correct, as if
> changes are possible in principle, then data/schema could change
> during the time QGIS (Desktop/Server) is open/running
>   * in environments other than development (testing, qa, production),
> database schema should never change without some clear procedure which
> should also include updating QGIS project files and testing them as
> required, therefore such re-checkings should never be required on
> testing/production environments.
>
>   I do understand that the current situation/understanding is
> different from mine, so in order to satisfy everybody, maybe it would
> be feasible to add say an environment variable like QGIS_FAST=TRUE (or
> QGIS_PRODUCTION=TRUE) and if this is set - re-checking queries could
> be skipped? We've been running modified QGIS without re-checking in
> production for months and are very happy with results. This should
> also improve the benchmarks of QGIS Server as it is known that initial
> opening of the project takes considerable amount of time.
>
>   What is your opinion?
>
>   Thank you
>
> --
> Tomas
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[QGIS-Developer] QGIS "fast" mode

[Tomas Straupis]

Hello

  I would like to know opinion of QGIS developer community on the
question of "self-fixing queries" in the context of database layers.

  When you add a new database layer, QGIS queries required data, asks
user for other information and then saves it in the project file.
  Now when you re-open the project, QGIS is re-querying a lot of data
which is already saved in the project: geometry type, srid, attributes
(table columns/types).
  The downsides are:
  * these "re-checking" queries take considerable time on large
databases: with millions of records from 1minute up to several hours,
database is tuned - WMS/WFS queries finish in ~100ms.
  * in QGIS server environment these re-checking queries are executed
each time when apache launches a new QGIS server process
  * QGIS might by itself (without human interaction) decide to  work
differently compared to what it was asked to do when the project was
created if it finds that "something has changed"
  * re-checking during project opening is not 100% correct, as if
changes are possible in principle, then data/schema could change
during the time QGIS (Desktop/Server) is open/running
  * in environments other than development (testing, qa, production),
database schema should never change without some clear procedure which
should also include updating QGIS project files and testing them as
required, therefore such re-checkings should never be required on
testing/production environments.

  I do understand that the current situation/understanding is
different from mine, so in order to satisfy everybody, maybe it would
be feasible to add say an environment variable like QGIS_FAST=TRUE (or
QGIS_PRODUCTION=TRUE) and if this is set - re-checking queries could
be skipped? We've been running modified QGIS without re-checking in
production for months and are very happy with results. This should
also improve the benchmarks of QGIS Server as it is known that initial
opening of the project takes considerable amount of time.

  What is your opinion?

  Thank you

-- 
Tomas
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Georeferencer improvements

[Luigi Pirelli]

inline

On Fri, 13 Mar 2020 at 16:13, Martin Dobias  wrote:

>
> 1. Georeferencer opens in a new window. In order to see the main map
> canvas and the georeferenced layer side by side, it is necessary to do
> some resizing of windows which is inconvenient. It would be maybe more
> practical to have the georeferenced image in a dock widget in the main
> window - and all georeferencer related map tools included in the main
> QGIS window as well. (Or maybe the other way around - to keep
> georeferencer as a separate window, but allow the main map canvas to
> be embedded in it?)
>

se j rectifier plugin

>
> 2. Georeferencer is a plugin. Its core/gui functionality cannot be
> reused elsewhere. It would be useful to have some Processing
> algorithms to load/save GCPs, to extract GCPs to raster images, to
> save GCPs to raster images and to warp imagery according to GCPs (and
> chosen transformation method).
>

yes please


>
> 3. Georeferencer does not allow live preview of how the georeferenced
> raster image would end up looking on the map - user is required to
> write warped image to disk as a new file and only then see the
> results. It should be possible to use the estimated transformation
> based on GCPs and show the layer in the main map canvas any time (if
> wanted).
>

see j rectifier plugin

4. It uses a combination of GDAL-based transformation methods
> (Polynomial of 1st/2nd/3rd order, thin plate spline) and some of its
> own (Helmert, Projective), for which it also requires GSL library as a
> dependency. In my opinion, QGIS georeferencer should leave all the
> math to GDAL which does a great job there. Moreover, I have some
> doubts about the methods that do not come from GDAL: the “Helmert”
> method is simply 2D affine transform (offset+scale+angle), while true
> Helmert transformation is 3D transformation with 7 parameters. Do we
> need it if GDAL’s 1st order polynomial transformation should do just
> that? And the “Projective” method which is a bit of a mystery to me.
> First of all, I am not sure in what situation it is useful and the
> documentation only says it is “a linear rotation and translation of
> coordinates” - which is a bit vague. It looks like a subset of RPC
> method supported by GDAL (although GDAL does not have estimation of
> RPC coefficients). Overall I would be for removal of these methods and
> stick to what GDAL offers.
>

+1 to all in gdal... btw seems gdal has some hidden or poorly documented
features. e.g geoid and gcp seems (form documentation) mutually exclusive,
but they are not (reading the code) and can be combined to reach better
accuracy.

Regards
> Martin
>

cheers

Luigi Pirelli

**
* LinkedIn: https://www.linkedin.com/in/luigipirelli
* Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
* GitHub: https://github.com/luipir
* Book: Mastering QGIS3 - 3rd Edition

* Hire a team: http://www.qcooperative.net
**
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Georeferencer improvements

[Luigi Pirelli]

Hi Martin

this was one element in my todo list months ago.
In some automatic processing workflow, sometimes is necessary to give a
straightforward interactive tool to facilitate rectification when
polinomial parameters are not enough to reach precision... fortunately,
struggling with gdal code, I reached enough precision 1.5m (for our
application) on WorldView images.

BTW in case wouldn't be enough, I provided a custom and simplified QGIS to
go straightforward to create GCPs.

A better approach I found, but not ported to 3.x, is the use of "J
Rectifier" [1] plugin. This plugin theoretically reorganise de GUI to allo
live georeferencing that facilitating a lot the work. The main problem to
port to 3.x is that have to be redesigned from scratch, having one big
class doing all!
IMHO from this plugin we can get some interesting ideas how to facilitate
the user interaction.

Luigi Pirelli

[1]  https://plugins.qgis.org/plugins/J_Rectifier/

**
* LinkedIn: https://www.linkedin.com/in/luigipirelli
* Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
* GitHub: https://github.com/luipir
* Book: Mastering QGIS3 - 3rd Edition

* Hire a team: http://www.qcooperative.net
**


On Fri, 13 Mar 2020 at 16:13, Martin Dobias  wrote:

> Hi all
>
> There has been some interest from a client to improve the
> georeferencer in QGIS to better suit their needs, mainly to support
> their workflow where they commonly georeference multiple raster images
> at the same time. So for example you pick a single ground control
> point (GCP) with XY(Z) map coordinates and in multiple images you
> would visually pick where that point is located in image coordinates.
> Ideally you want to see side-by-side the main map canvas and images
> you are georeferencing.
>
> With this kind of change, it is maybe a good time to have a look at
> what are the other issues with the georeferencer and address them as
> well. The budget is limited (as usual), so I cannot promise that
> anything mentioned in this thread would get implemented, but I am sure
> there are people around who have some ideas what could be improved. At
> some point later I would like to prepare a QEP that will be hopefully
> in line with the discussion here.
>
> Let me start with a list of current issues I see with georeferencer in
> addition to the lack of georeferencing of multiple images at once. The
> following is a mixture of UI/UX and technical bits:
>
> 1. Georeferencer opens in a new window. In order to see the main map
> canvas and the georeferenced layer side by side, it is necessary to do
> some resizing of windows which is inconvenient. It would be maybe more
> practical to have the georeferenced image in a dock widget in the main
> window - and all georeferencer related map tools included in the main
> QGIS window as well. (Or maybe the other way around - to keep
> georeferencer as a separate window, but allow the main map canvas to
> be embedded in it?)
>
> 2. Georeferencer is a plugin. Its core/gui functionality cannot be
> reused elsewhere. It would be useful to have some Processing
> algorithms to load/save GCPs, to extract GCPs to raster images, to
> save GCPs to raster images and to warp imagery according to GCPs (and
> chosen transformation method).
>
> 3. Georeferencer does not allow live preview of how the georeferenced
> raster image would end up looking on the map - user is required to
> write warped image to disk as a new file and only then see the
> results. It should be possible to use the estimated transformation
> based on GCPs and show the layer in the main map canvas any time (if
> wanted).
>
> 4. It uses a combination of GDAL-based transformation methods
> (Polynomial of 1st/2nd/3rd order, thin plate spline) and some of its
> own (Helmert, Projective), for which it also requires GSL library as a
> dependency. In my opinion, QGIS georeferencer should leave all the
> math to GDAL which does a great job there. Moreover, I have some
> doubts about the methods that do not come from GDAL: the “Helmert”
> method is simply 2D affine transform (offset+scale+angle), while true
> Helmert transformation is 3D transformation with 7 parameters. Do we
> need it if GDAL’s 1st order polynomial transformation should do just
> that? And the “Projective” method which is a bit of a mystery to me.
> First of all, I am not sure in what situation it is useful and the
> documentation only says it is “a linear rotation and translation of
> coordinates” - which is a bit vague. It looks like a subset of RPC
> method supported by GDAL (although GDAL does not have estimation of
> RPC coefficients). Overall I would be for removal of these methods and
> stick to what