Re: [QGIS-Developer] [Processing][Article] Evaluation of Effective Cognition for the QGIS Processing Modeler

[Paolo Cavallini]

Hi João,

Il 16/03/20 00:10, Nyall Dawson ha scritto:

>> Is open access:
>> https://www.mdpi.com/2076-3417/10/4/1446/htm

thanks a lot for the suggestion. I see a number of practical suggestions
here, at least some of them look easy to implement.
Cheers.
-- 
Paolo Cavallini - www.faunalia.eu
QGIS.ORG Chair:
http://planet.qgis.org/planet/user/28/tag/qgis%20board/
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Understanding plugin management in QGIS - exposing QgsPluginRegistry to Python?

[Alexander Bruy]

I'm with Nyall here. Once we started moving Processing classes in C++
it became much more stable than pure Python solution (even with tests).

As was said previously Python support is optional and from my point of view
it is much better to port remainings of the Plugin Manager to C++ and leave
only tiny wrapper for loading Python plugins.

пн, 16 бер. 2020 о 01:26 Nyall Dawson  пише:
>
> On Sun, 15 Mar 2020 at 21:50, Sebastian M. Ernst  
> wrote:
> >
> > Hi Borys, Nyall, Matthias,
> >
> > thanks all for your replies! Let me address some of the points/concerns
> > raised.
> >
> > > Actually, over time we've found that the opposite is true -- the
> > > Python code in QGIS master is the **most** work to maintain, and a
> > > constant source of regressions. That's due to the nature of the Python
> > > language itself, not a reflection of the quality of the code...
> >
> > Nyall, could you point me to some specific regressions that you are
> > referring to? Bugs or their IDs (so I can look them up)? When you say
> > "nature of the Python language", which specific aspects are you
> > referring to? I know there is a lot that one can legitimately criticize
> > about the language, but again I'd be highly interested in what
> > specifically is causing you headaches.
>
> The issues we see most often stem not from any particular use of
> Python, but (as Matthias has pointed out) through the nature of
> Pythons weak typing and lack of compile time checks. A simple example:
> if I alter the signature of a virtual method in one of the c++ classes
> to add an extra argument or change a return type, I'll know right away
> (well, when I next compile) if I've broken some existing code. If I do
> the same in Python, I'm basically trusting that my particular IDE can
> correctly identify ALL use of that method, because I won't get any
> warning if I did break something. Unit tests help, but will never
> completely avoid this kind of breakage.
>
> And we see this happen ALL. THE. TIME. in QGIS Python code. Processing
> (through NO fault of the maintainers) was a frequent source of
> breakage in QGIS 2.x, and it was only by porting the guts of it away
> from Python and to c++ that we could finally get things stable. Now
> it's happening all the time in dbmanager -- I don't think there's been
> a single recent QGIS release which hasn't had some form of regression
> in db manager. And again, I blame that on the nature of Python, not on
> the existing code or maintainers.
>
> Another HUGE factor with Python+QGIS is that the actual methods of
> creating the Python bindings + the PyQt library itself is a mysterious
> black box, full of weird inconsistencies, random results, memory
> mismanagement and crashy code practices. I shudder to think of the
> amount of time I've wasted fighting with SIP and its unpredictable
> behavior. You could write the BEST Python code the world has ever
> seen, using all the Python best practice coding techniques you want,
> and you'll still hit random regressions and issues due to these
> libraries. QGIS c++ code is just MUCH more predictable, transparent,
> and easier to debug.
>
> >> That's due to the nature of the Python
> > > language itself, not a reflection of the quality of the code...
> > Reading the code of the Python (environment) integration, I can see that
> > it is prone to all sorts of issues - but I'd be interested in your
> > experience in what actually causes problems the most.
>
> My point wasn't a comment of the quality of the Python code either
> way. Rather, my point was that **regardless** of the quality of the
> Python code, you'll still end up with a difficult to maintain, fragile
> result!
>
> > > That would be my preference too -- the overall movement in QGIS code
> > > is to reduce the overall amount of Python code in master, not the
> > > reverse.
> >
> > I can understand where you are coming from, however please allow me to
> > suggest the following: I would like to do what I already hinted on, i.e.
> > unify most plugin management code into a common, modern Python code base
> > with tests etc.
>
> This work sounds fantastic, and highly valuable to QGIS. It's GREAT to
> hear that you want to investigate this!!
>
> But please please please can I ask you to seriously reconsider your
> approach? I've been working on this project for nearly 10 years now,
> and my experience time and time again is that large blocks of Python
> code in QGIS end up unstable. Modern c++ + qt is really not that much
> trickier to write then Python, and if you include good test coverage,
> then the code becomes almost self-maintaining*.
>
> Python is a GREAT choice for QGIS scripts and addons, but I just don't
> think it's a suitable choice here.
>
> Nyall
>
> * at least, until a major qt release deprecates stuff you rely on! :P
>
>
>
> I would like to show how this could look like. You do
> > not have to accept it or merge it. After all, I would probably have to
> > write an enhancement proposal first

Re: [QGIS-Developer] Understanding plugin management in QGIS - exposing QgsPluginRegistry to Python?

[Paolo Cavallini]

Hi Sebastian,
thanks for your thoughts.

Il 15/03/20 22:31, Sebastian M. Ernst ha scritto:
> 
> Well, the bottom line is more efficiently tapping into pypi and conda
> packages. QGIS / OSGeo4W pretty much acts as a rather limited Python
> distribution. It's missing a lot of interesting scientific packages that
> currently see rapid development. Their distribution as part of QGIS /
> OSGeo4W would probably go much beyond the scope and resources of the
> QGIS project. If, on the other hand, QGIS was "simply" yet another
> C++-heavy Python package within a larger ecosystem, let's say Anaconda,
> this would add an interesting new perspective. I have been chasing the
> idea that QGIS plugins (more specifically their meta data) are lacking a
> lot of features of Python wheels or Conda packages. They really can not
> carry dependencies (other than dependencies to other plugins, added in
> QGIS 3.6 I believe). It's actually officially recommended to make Python
> dependencies part of QGIS plugins' distribution files (c & p), which can
> be a no-go for a number of good reasons. If you imagine that QGIS
> plugins are instead (or additionally) distributed as conda packages
> (with all of conda's package management features) while the entire thing
> remains manageable through a simple GUI, the QGIS plugin manager, I can
> see some interesting new use-cases and possibilities (especially for
> non-technical and Windows users). This is more or less the gist of my
> thinking.

having a proper way of managing plugin dependencies would be a huge
improvement, as this is one of the weakest points of our plugin management.
Cheers.
-- 
Paolo Cavallini - www.faunalia.eu
QGIS.ORG Chair:
http://planet.qgis.org/planet/user/28/tag/qgis%20board/
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Requirements for new contributors

[Paolo Cavallini]

Thank you Morten, welcome on board!
Cheers.

Il 15/03/20 20:48, Morten Sickel ha scritto:
>  
> 
> I hereby confirm,
> 
>  
> 
> I have read, understood and agree to the code of conduct, the coding
> standard and the diversety statements as described on
> https://qgis.org/en/site/getinvolved/development/contributor_requirements.html
> 
>  
> 
> Morten Sickel
> 
> mor...@sickel.net 
> 
>  
> 
> I hope to be able to participate a bit on the python side – I have had
> some problems with the db manager. I am going to do a bit more
> investigation to write a bug report, but I think I have an idea how to
> fix the problem.
> 
>  
> 
> Morten
> 
> 
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> 

-- 
Paolo Cavallini - www.faunalia.eu
QGIS.ORG Chair:
http://planet.qgis.org/planet/user/28/tag/qgis%20board/
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Understanding plugin management in QGIS - exposing QgsPluginRegistry to Python?

[Nyall Dawson]

On Sun, 15 Mar 2020 at 21:50, Sebastian M. Ernst  wrote:
>
> Hi Borys, Nyall, Matthias,
>
> thanks all for your replies! Let me address some of the points/concerns
> raised.
>
> > Actually, over time we've found that the opposite is true -- the
> > Python code in QGIS master is the **most** work to maintain, and a
> > constant source of regressions. That's due to the nature of the Python
> > language itself, not a reflection of the quality of the code...
>
> Nyall, could you point me to some specific regressions that you are
> referring to? Bugs or their IDs (so I can look them up)? When you say
> "nature of the Python language", which specific aspects are you
> referring to? I know there is a lot that one can legitimately criticize
> about the language, but again I'd be highly interested in what
> specifically is causing you headaches.

The issues we see most often stem not from any particular use of
Python, but (as Matthias has pointed out) through the nature of
Pythons weak typing and lack of compile time checks. A simple example:
if I alter the signature of a virtual method in one of the c++ classes
to add an extra argument or change a return type, I'll know right away
(well, when I next compile) if I've broken some existing code. If I do
the same in Python, I'm basically trusting that my particular IDE can
correctly identify ALL use of that method, because I won't get any
warning if I did break something. Unit tests help, but will never
completely avoid this kind of breakage.

And we see this happen ALL. THE. TIME. in QGIS Python code. Processing
(through NO fault of the maintainers) was a frequent source of
breakage in QGIS 2.x, and it was only by porting the guts of it away
from Python and to c++ that we could finally get things stable. Now
it's happening all the time in dbmanager -- I don't think there's been
a single recent QGIS release which hasn't had some form of regression
in db manager. And again, I blame that on the nature of Python, not on
the existing code or maintainers.

Another HUGE factor with Python+QGIS is that the actual methods of
creating the Python bindings + the PyQt library itself is a mysterious
black box, full of weird inconsistencies, random results, memory
mismanagement and crashy code practices. I shudder to think of the
amount of time I've wasted fighting with SIP and its unpredictable
behavior. You could write the BEST Python code the world has ever
seen, using all the Python best practice coding techniques you want,
and you'll still hit random regressions and issues due to these
libraries. QGIS c++ code is just MUCH more predictable, transparent,
and easier to debug.

>> That's due to the nature of the Python
> > language itself, not a reflection of the quality of the code...
> Reading the code of the Python (environment) integration, I can see that
> it is prone to all sorts of issues - but I'd be interested in your
> experience in what actually causes problems the most.

My point wasn't a comment of the quality of the Python code either
way. Rather, my point was that **regardless** of the quality of the
Python code, you'll still end up with a difficult to maintain, fragile
result!

> > That would be my preference too -- the overall movement in QGIS code
> > is to reduce the overall amount of Python code in master, not the
> > reverse.
>
> I can understand where you are coming from, however please allow me to
> suggest the following: I would like to do what I already hinted on, i.e.
> unify most plugin management code into a common, modern Python code base
> with tests etc.

This work sounds fantastic, and highly valuable to QGIS. It's GREAT to
hear that you want to investigate this!!

But please please please can I ask you to seriously reconsider your
approach? I've been working on this project for nearly 10 years now,
and my experience time and time again is that large blocks of Python
code in QGIS end up unstable. Modern c++ + qt is really not that much
trickier to write then Python, and if you include good test coverage,
then the code becomes almost self-maintaining*.

Python is a GREAT choice for QGIS scripts and addons, but I just don't
think it's a suitable choice here.

Nyall

* at least, until a major qt release deprecates stuff you rely on! :P



I would like to show how this could look like. You do
> not have to accept it or merge it. After all, I would probably have to
> write an enhancement proposal first. So, please take it as a proof of
> concept for now. How about that?
>
> Once I have a working proof of concept, I actually have a few ideas on
> how to take it forward and add a few interesting features to it - again
> as proof of concept. One key idea is the integration of the conda
> package manager. Other than pip, it actually has an official, stable
> API. And, well, QGIS has recently been packaged for Anaconda for all
> relevant platforms (kudos and thanks to everyone involved!). A conda
> integration has a hell of a lot of potential.
>
> > I admit 

Re: [QGIS-Developer] [Processing][Article] Evaluation of Effective Cognition for the QGIS Processing Modeler

[Nyall Dawson]

On Mon, 16 Mar 2020 at 08:29, João Gaspar  wrote:
>
> Dear devs,
>
> While I was doing article research and I found this article about Processing 
> Modeler that was published in February 2020.
>
> I don't know if one of you were aware of this or if this can help to improve 
> the Processing Modeler. I thought this can be useful that was the reason to 
> share with you.
>
> This is the abstract:
> This article presents an evaluation of the QGIS Processing Modeler from the 
> point of view of effective cognition. The QGIS Processing Modeler uses visual 
> programming language for workflow design. The functionalities of the visual 
> component and the visual vocabulary (set of symbols and line connectors) are 
> both important. The form of symbols affects how workflow diagrams may be 
> understood. The article discusses the results of assessing the Processing 
> Modeler’s visual vocabulary in QGIS according to the Physics of Notations 
> theory. The article evaluates visual vocabularies from the older QGIS 2.x and 
> newer 3.x versions. The paper identifies serious design flaws in the 
> Processing Modeler. Applying the Physics of Notations theory resulted in 
> certain practical recommendations, such as changing the fill colour of 
> symbols, increasing the size and variety of inner icons, removing functional 
> icons, and using a straight connector line instead of a curved line. Another 
> recommendation was to provide a supplemental preview window for the entire 
> model in order to improve user navigation in huge models. Objective 
> eye-tracking measurements validated some results of the evaluation using the 
> Physics of Notations. The respondents read workflows to solve different tasks 
> and their gazes were tracked. Evaluation of the eye-tracking metrics revealed 
> the respondents’ reading patterns of the diagram. Evaluation using both 
> Physics of Notation theory and eye-tracking measurements inspired 
> recommendations for improving visual notation. A set of recommendations for 
> users is also given, which can be applied easily in practice using a 
> contemporary visual notation.

Really cool! (Wonder if we could convince the authors to extend the
study to other parts of QGIS...!)

Will respond properly when I've had a chance to read and digest fully.

Nyall

>
> Is open access:
> https://www.mdpi.com/2076-3417/10/4/1446/htm
>
> Best regards,
> João Gaspar aka James
>
>
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[QGIS-Developer] [Processing][Article] Evaluation of Effective Cognition for the QGIS Processing Modeler

[João Gaspar]

Dear devs,

While I was doing article research and I found this article about
Processing Modeler that was published in February 2020.

I don't know if one of you were aware of this or if this can help to
improve the Processing Modeler. I thought this can be useful that was the
reason to share with you.

This is the abstract:
*This article presents an evaluation of the QGIS Processing Modeler from
the point of view of effective cognition. The QGIS Processing Modeler uses
visual programming language for workflow design. The functionalities of the
visual component and the visual vocabulary (set of symbols and line
connectors) are both important. The form of symbols affects how workflow
diagrams may be understood. The article discusses the results of assessing
the Processing Modeler’s visual vocabulary in QGIS according to the Physics
of Notations theory. The article evaluates visual vocabularies from the
older QGIS 2.x and newer 3.x versions. The paper identifies serious design
flaws in the Processing Modeler. Applying the Physics of Notations theory
resulted in certain practical recommendations, such as changing the fill
colour of symbols, increasing the size and variety of inner icons, removing
functional icons, and using a straight connector line instead of a curved
line. Another recommendation was to provide a supplemental preview window
for the entire model in order to improve user navigation in huge models.
Objective eye-tracking measurements validated some results of the
evaluation using the Physics of Notations. The respondents read workflows
to solve different tasks and their gazes were tracked. Evaluation of the
eye-tracking metrics revealed the respondents’ reading patterns of the
diagram. Evaluation using both Physics of Notation theory and eye-tracking
measurements inspired recommendations for improving visual notation. A set
of recommendations for users is also given, which can be applied easily in
practice using a contemporary visual notation.*

Is open access:
https://www.mdpi.com/2076-3417/10/4/1446/htm

Best regards,
João Gaspar aka James
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Understanding plugin management in QGIS - exposing QgsPluginRegistry to Python?

[Sebastian M. Ernst]

Hi Matthias,

> Have a look at all the issues on db manager for example. Many of them
> are caused by duck-typing, interfaces not being implemented (properly)
> by subclasses, pieces that have been forgotten during refactoring ...

thanks for the directions, I will look into it.

> I would like it a lot to get a couple of pull requests that target such
> weaknesses.
> Such pull requests are normally accepted quite quickly and much
> appreciated.
> [...]
> No one is in the position to say no to this, the GPL grants you the
> rights for that :)

I'd issue pull requests if it was an easy repair. I was actually
starting to poke around because I thought I could add a feature. I ended
up thinking about a significant consolidation if not a rewrite (before
even considering adding features) because I think the code really needs
it. So before I begin to send pull requests your way, I decided to
explore what kind of changes you guys would be willing to accept
(short/long term) before I invest more time into it. If a re-write is
the lesser evil, this opens a bunch of interesting options /
opportunities, I'd say, and then it makes a lot of sense to ask for QGIS
developers' opinions.

> For your information: next to the plugin manager, there is also the
> Python console [...]

I have noticed ;) There are a few issues in the console that keep
annoying me, but my current focus is on the plugin system.

> This would be a very nice addition indeed.
> 
> Plans in this direction have been on our wish list for this year. We
> have just had some discussions with Olivier about this this week.
> 
> This topic would deserve it's own QEP and I'm very interested in knowing
> a bit more about what you are after here.

Well, the bottom line is more efficiently tapping into pypi and conda
packages. QGIS / OSGeo4W pretty much acts as a rather limited Python
distribution. It's missing a lot of interesting scientific packages that
currently see rapid development. Their distribution as part of QGIS /
OSGeo4W would probably go much beyond the scope and resources of the
QGIS project. If, on the other hand, QGIS was "simply" yet another
C++-heavy Python package within a larger ecosystem, let's say Anaconda,
this would add an interesting new perspective. I have been chasing the
idea that QGIS plugins (more specifically their meta data) are lacking a
lot of features of Python wheels or Conda packages. They really can not
carry dependencies (other than dependencies to other plugins, added in
QGIS 3.6 I believe). It's actually officially recommended to make Python
dependencies part of QGIS plugins' distribution files (c & p), which can
be a no-go for a number of good reasons. If you imagine that QGIS
plugins are instead (or additionally) distributed as conda packages
(with all of conda's package management features) while the entire thing
remains manageable through a simple GUI, the QGIS plugin manager, I can
see some interesting new use-cases and possibilities (especially for
non-technical and Windows users). This is more or less the gist of my
thinking.

>> If I want to develop my proof of concept (and actually proof that this
>> could work), I'd ideally need access to at least some methods from
>> `QgsPluginRegistry`. Back to my original question: Can this class be
>> accessed from Python? Or could someone help me with exMy suggestion
>> would posing the
>> relevant methods (from one/the relevant instance of this class) to
>> Python? I have no trouble with building QGIS from source if required
> 
> No. It is in app, and the app library is not exposed to Python.

How complicated is it to change that, let's say for just
`QgsPluginRegistry`, with a really ugly patch perhaps (that is not meant
to go into master)?

Best regards,
Sebastian
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Understanding plugin management in QGIS - exposing QgsPluginRegistry to Python?

[Matthias Kuhn]

Hi Sebastian

On 3/15/20 12:50 PM, Sebastian M. Ernst wrote:

Hi Borys, Nyall, Matthias,

thanks all for your replies! Let me address some of the points/concerns
raised.


Actually, over time we've found that the opposite is true -- the
Python code in QGIS master is the **most** work to maintain, and a
constant source of regressions. That's due to the nature of the Python
language itself, not a reflection of the quality of the code...

Nyall, could you point me to some specific regressions that you are
referring to? Bugs or their IDs (so I can look them up)? When you say
"nature of the Python language", which specific aspects are you
referring to? I know there is a lot that one can legitimately criticize
about the language, but again I'd be highly interested in what
specifically is causing you headaches.


Have a look at all the issues on db manager for example. Many of them 
are caused by duck-typing, interfaces not being implemented (properly) 
by subclasses, pieces that have been forgotten during refactoring ...


I'm sure we will find explanations for each of them why they were there 
and how they could have been prevented. Very often the answer will be 
"unit tests missing".


While "unit tests" are also good for C++, it does by design already 
force a good amount of sanity checks onto code at compile time.




Reading the code of the Python (environment) integration, I can see that
it is prone to all sorts of issues - but I'd be interested in your
experience in what actually causes problems the most.

As for the code quality ... I do not want to offend anyone, seriously.
*PLEASE* take it as constructive criticism. I have been heavily
consulting on Python development for the past couple of years, so I have
reviewed, debugged and refactored a lot of code. One aspect that
immediately struck me when reading QGIS' Python code was the fact that
there were (absolutely unnecessary) relics of an automatic "2to3"
conversion all over the place. This tells a lot about the state of the
code already. Beyond that, I have found a few really bizarre (and
outright dangerous) uses of the "global" keyword. In a few cases for
error handling, I believe ... Yes, there are (few) legitimate usecases
for "global" in Python, but as far as my findings in QGIS go, none of
those make any sense. This can be solved in a much cleaner way by
exploiting name spaces and "mutable variables" (and exceptions). Having
said that, most of the code cares very little about mutable data types.
A few calls to "copy" methods in interesting places tell a story of
running into issues related to mutability without caring too much about
it. Well, I really do not want to insinuate a lack of understanding.
Also, going into the same direction, the code heavily relies on the
garbage collector doing a really good job. If this was performance
critical, it would be very slow. If someone relying on the correct use
of mutability would use some of the Python APIs in QGIS, it would sure
as hell lead to layers of bugs. As far as style goes, it is really hard
to see what internal and what external APIs are (i.e. which methods are
only called from within classes/modules or from the outside). Because
there is no public/private in Python, the typical convention is to use
(single) leading underscores in names. It is a minor detail but one that
leads to all sorts of issues when maintaining code in the longer run. A
somewhat bigger style issue is the fact that there is almost no
exception raising/handling as well as no parameter type, bounds and
consistency checking in functions/methods. Again, as long as you do not
touch this code (and stars are aligned in the right way), that's not an
issue. It is only hitting you on the longer run.

Ok, let's stop here. I really hope I have not trashed this code too much
... I just want to underline that (a lack of) code quality most
definitely adds to your negative experience with Python in QGIS (core)
from my perspective.


You certainly have some valid points here.

Anything that helps to improve the current situation of (python) code 
will be appreciated!


I would like it a lot to get a couple of pull requests that target such 
weaknesses.


Such pull requests are normally accepted quite quickly and much appreciated.

For your information: next to the plugin manager, there is also the 
Python console which in contrast to the Plugin manager is very unlikely 
to be replaced by C++ code. Any fixes in there will be long lasting and 
will help to strengthen the code for the upcoming years.





That would be my preference too -- the overall movement in QGIS code
is to reduce the overall amount of Python code in master, not the
reverse.

I can understand where you are coming from, however please allow me to
suggest the following: I would like to do what I already hinted on, i.e.
unify most plugin management code into a common, modern Python code base
with tests etc. I would like to show how this could look like. You do
not have to a

[QGIS-Developer] Requirements for new contributors

[Morten Sickel]

I hereby confirm,

I have read, understood and agree to the code of conduct, the coding standard 
and the diversety statements as described on 
https://qgis.org/en/site/getinvolved/development/contributor_requirements.html

Morten Sickel
mor...@sickel.net

I hope to be able to participate a bit on the python side – I have had some 
problems with the db manager. I am going to do a bit more investigation to 
write a bug report, but I think I have an idea how to fix the problem.

Morten
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Understanding plugin management in QGIS - exposing QgsPluginRegistry to Python?

[Borys Jurgiel]

Dnia niedziela, 15 marca 2020 12:50:04 CET Sebastian M. Ernst pisze:

> As far as I can
> see it, C++ plugins have to be compiled with QGIS anyway. At runtime,
> all a user can do is to activate or deactivate them. Borys, am I
> understanding this correctly? 

Yes

> If a *minimal* C++ GUI would allow exactly
> that, you could reduce a lot of the complexity. The rest could
> (following my instinct) cleanly be written in Python. I.e. if (and only
> if) Python is present, a fully featured plugin manager (for C++ *and*
> Python plugins) could drop in and *replace* the simple C++ plugin on/off
> manager GUI. How about that?

It would work, too.

I don't have any strong opinion about the two opposite directions and don't 
want to take the floor, as I probably won't be able to contribute. I was 
thinking about refactoring the Installer (what I terribly wrote being a 
newbie) as well as rewritting the Manager with QtQuick for years, but never 
found enough time for quite a big task.

Regards,

Borys



___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Doc Travis

[Alessandro Pasotti]

On Sun, Mar 15, 2020 at 4:45 PM Richard Duivenvoorde
 wrote:
>
> On 3/15/20 4:08 PM, Richard Duivenvoorde wrote:
> > We will probably change master branch soon so it will get build in the
> > same way...
>
> Done:
>
> https://docs.qgis.org/testing/en/docs/
>
> Please have a look and fix things.
> I've barely looked at old build versions, just copied build machinery
> from qgisdoc2020 branch. So if something was kicked into eternity,
> please resurrect if from git...


Hi Richard,

looks great!

>
> We still need to sort out how to build automaticaly: travis, github
> actions, whatever. Please feel free to kick in!
>


I might have missed something, but if I'm not wrong Travis already
build the docs during the testing process (but in a single language?).

What you are talking about is continuous deployment (CD) right?

You could probably do with both Travis and github workflows, you just
need to make sure that the CD only happens on each commit to master.

In order to be able to push the build to the deployment server you'll
probably need a deployment SSH key stored as a secret on Travis (or GH
workflow) that allow the Travis (or GH workflow) process to transfer
the built files to the production server.


--
Alessandro Pasotti
w3:   www.itopen.it
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Doc Travis

[Richard Duivenvoorde]

On 3/15/20 4:08 PM, Richard Duivenvoorde wrote:
> We will probably change master branch soon so it will get build in the
> same way...

Done:

https://docs.qgis.org/testing/en/docs/

Please have a look and fix things.
I've barely looked at old build versions, just copied build machinery
from qgisdoc2020 branch. So if something was kicked into eternity,
please resurrect if from git...

We still need to sort out how to build automaticaly: travis, github
actions, whatever. Please feel free to kick in!

Regards,

Richard
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Doc Travis

[Richard Duivenvoorde]

On 3/15/20 3:37 PM, r.m.aguilardearch...@utwente.nl wrote:
> Hey. Guys. 
> Just to let you know. I built the doc in Windows. Looks beautiful. 
> I just felt a bit confusing with the windows instructions. 
> Because when I cloned the repo I did not get the make.bat ( I could not clone 
> the qgisdoc2020 branch directly- beginner mistake?)
> So, I needed to create the branch first before actually compile it. 
> Perhaps adding such make.bat in the master branch may help? 

Hi Rosa,

Till now the qgis2020 branch is just an experiment to build 3.10, that
is. At this moment there was just no make.bat on master yet  :-)

We choose to build 3.10 because all the translation machinery has to be
there and working too.

We will probably change master branch soon so it will get build in the
same way...

If others find errors/issues with instructions, please fix.

Regards,

Richard

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Doc Travis

[r.m.aguilardearch...@utwente.nl]

Hey. Guys. 
Just to let you know. I built the doc in Windows. Looks beautiful. 
I just felt a bit confusing with the windows instructions. 
Because when I cloned the repo I did not get the make.bat ( I could not clone 
the qgisdoc2020 branch directly- beginner mistake?)
So, I needed to create the branch first before actually compile it. 
Perhaps adding such make.bat in the master branch may help? 
Nice work. 

Cheers, 
Rosa 

-
MSc Rosa Aguilar
PhD Candidate
Faculty of Geoinformation Science and Earth Observation
University of Twente
Enschede, The Netherlands
Telephone: +31 647616029
https://www.linkedin.com/in/rosamaguilar/?locale=en_US
https://rosaguilar.github.io/
-

-Original Message-
From: QGIS-Developer  On Behalf Of 
Richard Duivenvoorde
Sent: 15 March 2020 09:44
To: matteo ; qgis-developer@lists.osgeo.org
Subject: Re: [QGIS-Developer] QGIS Doc Travis

On 3/15/20 8:57 AM, matteo wrote:
> That's what comes into my mind:
> 
> * verify Transifex pulling/pushing
> * apache on the server to publish the docs
> * ... what else?

@Raymond thanks for organising the hackfest! Sorry for not being there... But 
the 'doc-team' did some things related to documentations because there WAS a 
hackfest \o/

There are 2 things I want to discuss here.

First our main goal: "KISS".
Everybody on windows or linux should be able to build at least the english docs.

Second: stick to Sphinx and it's tools as much as possible (e.g. instead of 
custom scripts: use sphinx-intl to create transifex stuff).

My main 2 questions, I would like to discuss:

-1- CAN we use Travis to build the translated site (daily) (technically:
yes, I know, but...)

-2- WHICH process should be responsible for the pulling of the transifex 
translations into the build process

Ad -1-
"Make doctest" will use the qgis/qgis docker container to build and run the 
tests. Travis is happy with that [4]

So we could update or base a new image on it containing all language related 
stuff, and use travis to build all languages. Current 3.4 builds pull fresh 
from transifex (making a language build lasts for half an hour now 
sometimes...) If we push the po files into github, it will only be 10 minutes.
But my main question is: can/should we 'misuse' travis to build all those 
(currently 18) languages daily.

Ad -2-
Several options:
- during our main build, the build pulls (tx pull) from transifex. Is what we 
do now: Transifex is main repo for translations. IF an individual wants to 
build a translated doc, she/he needs to pull the translations from transifex 
itself (which is a disadvantage, as pulling translations from tx is SLOW...)
- we run a daily script which pulls the translations from tx and commits those 
po files (for ALL languages to be build) in git-repo.
Main advantage: we keep being boss over the po files ourselves, AND everybody 
can build translated docs, because the po files are (well 1 day old...) in git
- use the Transifex Github integration (I played with it one year ago, and then 
it could not handle our long resource handles at that time...)

I think I prefer the second or third option, will push nl, it and fr 
translations into git later today (by just pulling the 3.4 translations from 
build server), so everybody can try to build translated sites too.
Als together with maybe building it on travis, it is probably the best way..

Want to see current status?

- vanilla Sphinx local ( EVERYBODY can try to build branch qgisdoc2020 NOW on 
windows and linux, see README [0] )

- vanilla Sphinx remote ('make site' works if you can like me rsync to
qgis2 [1]) (NOTE: a lot has to be done there)

- everybody should now be able to 'just edit' on github [2]

- 'make doctest' (== building in te qgis/qgis docker image) is also working, 
see travis [4] qgisdoc2020 branch there


Sorry for being rather long, and while writing maybe answering my own questions 
:-)

But I really really hope we can keep stuff as simple as possible and encourage 
everybody to JOIN. Please contact us via documentation list

About PDF's: maybe later... but I'm OK without those...
So if you want them: join us

Regards,

Richard Duivenvoorde


[0] https://github.com/qgis/QGIS-Documentation/tree/qgisdoc2020
[1] https://docs.qgis.org/3.10/
[3]
https://github.com/qgis/QGIS-Documentation/blob/qgisdoc2020/docs/user_manual/introduction/getting_started.rst
[4] https://travis-ci.org/github/qgis/QGIS-Documentation
[5] https://docs.transifex.com/transifex-github-integrations/github-tx-ui
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://l

Re: [QGIS-Developer] Understanding plugin management in QGIS - exposing QgsPluginRegistry to Python?

[Sebastian M. Ernst]

Hi Borys, Nyall, Matthias,

thanks all for your replies! Let me address some of the points/concerns
raised.

> Actually, over time we've found that the opposite is true -- the
> Python code in QGIS master is the **most** work to maintain, and a
> constant source of regressions. That's due to the nature of the Python
> language itself, not a reflection of the quality of the code...

Nyall, could you point me to some specific regressions that you are
referring to? Bugs or their IDs (so I can look them up)? When you say
"nature of the Python language", which specific aspects are you
referring to? I know there is a lot that one can legitimately criticize
about the language, but again I'd be highly interested in what
specifically is causing you headaches.

Reading the code of the Python (environment) integration, I can see that
it is prone to all sorts of issues - but I'd be interested in your
experience in what actually causes problems the most.

As for the code quality ... I do not want to offend anyone, seriously.
*PLEASE* take it as constructive criticism. I have been heavily
consulting on Python development for the past couple of years, so I have
reviewed, debugged and refactored a lot of code. One aspect that
immediately struck me when reading QGIS' Python code was the fact that
there were (absolutely unnecessary) relics of an automatic "2to3"
conversion all over the place. This tells a lot about the state of the
code already. Beyond that, I have found a few really bizarre (and
outright dangerous) uses of the "global" keyword. In a few cases for
error handling, I believe ... Yes, there are (few) legitimate usecases
for "global" in Python, but as far as my findings in QGIS go, none of
those make any sense. This can be solved in a much cleaner way by
exploiting name spaces and "mutable variables" (and exceptions). Having
said that, most of the code cares very little about mutable data types.
A few calls to "copy" methods in interesting places tell a story of
running into issues related to mutability without caring too much about
it. Well, I really do not want to insinuate a lack of understanding.
Also, going into the same direction, the code heavily relies on the
garbage collector doing a really good job. If this was performance
critical, it would be very slow. If someone relying on the correct use
of mutability would use some of the Python APIs in QGIS, it would sure
as hell lead to layers of bugs. As far as style goes, it is really hard
to see what internal and what external APIs are (i.e. which methods are
only called from within classes/modules or from the outside). Because
there is no public/private in Python, the typical convention is to use
(single) leading underscores in names. It is a minor detail but one that
leads to all sorts of issues when maintaining code in the longer run. A
somewhat bigger style issue is the fact that there is almost no
exception raising/handling as well as no parameter type, bounds and
consistency checking in functions/methods. Again, as long as you do not
touch this code (and stars are aligned in the right way), that's not an
issue. It is only hitting you on the longer run.

Ok, let's stop here. I really hope I have not trashed this code too much
... I just want to underline that (a lack of) code quality most
definitely adds to your negative experience with Python in QGIS (core)
from my perspective.

> That would be my preference too -- the overall movement in QGIS code
> is to reduce the overall amount of Python code in master, not the
> reverse.

I can understand where you are coming from, however please allow me to
suggest the following: I would like to do what I already hinted on, i.e.
unify most plugin management code into a common, modern Python code base
with tests etc. I would like to show how this could look like. You do
not have to accept it or merge it. After all, I would probably have to
write an enhancement proposal first. So, please take it as a proof of
concept for now. How about that?

Once I have a working proof of concept, I actually have a few ideas on
how to take it forward and add a few interesting features to it - again
as proof of concept. One key idea is the integration of the conda
package manager. Other than pip, it actually has an official, stable
API. And, well, QGIS has recently been packaged for Anaconda for all
relevant platforms (kudos and thanks to everyone involved!). A conda
integration has a hell of a lot of potential.

> I admit the the infrastructure became a bit complicated ;) However, porting 
> the Plugin Manager to Python would require making Python a hard dependency of 
> QGIS GUI (or we should drop C++ plugins by porting them all to core, what is 
> discussed from time to time and doesn't seem to may happen).

Ok, so Python has to remain optional. This is just fine. As far as I can
see it, C++ plugins have to be compiled with QGIS anyway. At runtime,
all a user can do is to activate or deactivate them. Borys, am I
understanding t

[QGIS-Developer] Plugin [761] Clip Multiple Layers approval notification.

[noreply]

Plugin Clip Multiple Layers approval by zimbogisgeek.
The plugin version "[761] Clip Multiple Layers 3.1.0" is now approved
Link: http://plugins.qgis.org/plugins/ClipMultipleLayers/
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Understanding plugin management in QGIS - exposing QgsPluginRegistry to Python?

[Borys Jurgiel]

Dnia niedziela, 15 marca 2020 00:17:51 CET Matthias Kuhn pisze:
> Hi Sebastian, Borys,
> 
> What about the opposite way: moving things to C++ and leaving only a
> very thing wrapper for interacting with the plugins in place?

Well, it would be the cleanest option, if there are resources. The Installer 
code is the first to be rewritten anyway, so it would fix it at one go. 

Regards,

Borys



___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Doc Travis

[Alexandre Neto]

Hi Richard,

Instead of travis, maybe we can try to use github actions instead?

Alex

On Sun, Mar 15, 2020 at 8:43 AM Richard Duivenvoorde 
wrote:

> On 3/15/20 8:57 AM, matteo wrote:
> > That's what comes into my mind:
> >
> > * verify Transifex pulling/pushing
> > * apache on the server to publish the docs
> > * ... what else?
>
> @Raymond thanks for organising the hackfest! Sorry for not being
> there... But the 'doc-team' did some things related to documentations
> because there WAS a hackfest \o/
>
> There are 2 things I want to discuss here.
>
> First our main goal: "KISS".
> Everybody on windows or linux should be able to build at least the
> english docs.
>
> Second: stick to Sphinx and it's tools as much as possible (e.g. instead
> of custom scripts: use sphinx-intl to create transifex stuff).
>
> My main 2 questions, I would like to discuss:
>
> -1- CAN we use Travis to build the translated site (daily) (technically:
> yes, I know, but...)
>
> -2- WHICH process should be responsible for the pulling of the transifex
> translations into the build process
>
> Ad -1-
> "Make doctest" will use the qgis/qgis docker container to build and run
> the tests. Travis is happy with that [4]
>
> So we could update or base a new image on it containing all language
> related stuff, and use travis to build all languages. Current 3.4 builds
> pull fresh from transifex (making a language build lasts for half an
> hour now sometimes...) If we push the po files into github, it will only
> be 10 minutes.
> But my main question is: can/should we 'misuse' travis to build all
> those (currently 18) languages daily.
>
> Ad -2-
> Several options:
> - during our main build, the build pulls (tx pull) from transifex. Is
> what we do now: Transifex is main repo for translations. IF an
> individual wants to build a translated doc, she/he needs to pull the
> translations from transifex itself (which is a disadvantage, as pulling
> translations from tx is SLOW...)
> - we run a daily script which pulls the translations from tx and commits
> those po files (for ALL languages to be build) in git-repo.
> Main advantage: we keep being boss over the po files ourselves, AND
> everybody can build translated docs, because the po files are (well 1
> day old...) in git
> - use the Transifex Github integration (I played with it one year ago,
> and then it could not handle our long resource handles at that time...)
>
> I think I prefer the second or third option, will push nl, it and fr
> translations into git later today (by just pulling the 3.4 translations
> from build server), so everybody can try to build translated sites too.
> Als together with maybe building it on travis, it is probably the best
> way..
>
> Want to see current status?
>
> - vanilla Sphinx local ( EVERYBODY can try to build branch qgisdoc2020
> NOW on windows and linux, see README [0] )
>
> - vanilla Sphinx remote ('make site' works if you can like me rsync to
> qgis2 [1]) (NOTE: a lot has to be done there)
>
> - everybody should now be able to 'just edit' on github [2]
>
> - 'make doctest' (== building in te qgis/qgis docker image) is also
> working, see travis [4] qgisdoc2020 branch there
>
>
> Sorry for being rather long, and while writing maybe answering my own
> questions :-)
>
> But I really really hope we can keep stuff as simple as possible and
> encourage everybody to JOIN. Please contact us via documentation list
>
> About PDF's: maybe later... but I'm OK without those...
> So if you want them: join us
>
> Regards,
>
> Richard Duivenvoorde
>
>
> [0] https://github.com/qgis/QGIS-Documentation/tree/qgisdoc2020
> [1] https://docs.qgis.org/3.10/
> [3]
>
> https://github.com/qgis/QGIS-Documentation/blob/qgisdoc2020/docs/user_manual/introduction/getting_started.rst
> [4] https://travis-ci.org/github/qgis/QGIS-Documentation
> [5] https://docs.transifex.com/transifex-github-integrations/github-tx-ui
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Doc Travis

[Richard Duivenvoorde]

On 3/15/20 8:57 AM, matteo wrote:
> That's what comes into my mind:
> 
> * verify Transifex pulling/pushing
> * apache on the server to publish the docs
> * ... what else?

@Raymond thanks for organising the hackfest! Sorry for not being
there... But the 'doc-team' did some things related to documentations
because there WAS a hackfest \o/

There are 2 things I want to discuss here.

First our main goal: "KISS".
Everybody on windows or linux should be able to build at least the
english docs.

Second: stick to Sphinx and it's tools as much as possible (e.g. instead
of custom scripts: use sphinx-intl to create transifex stuff).

My main 2 questions, I would like to discuss:

-1- CAN we use Travis to build the translated site (daily) (technically:
yes, I know, but...)

-2- WHICH process should be responsible for the pulling of the transifex
translations into the build process

Ad -1-
"Make doctest" will use the qgis/qgis docker container to build and run
the tests. Travis is happy with that [4]

So we could update or base a new image on it containing all language
related stuff, and use travis to build all languages. Current 3.4 builds
pull fresh from transifex (making a language build lasts for half an
hour now sometimes...) If we push the po files into github, it will only
be 10 minutes.
But my main question is: can/should we 'misuse' travis to build all
those (currently 18) languages daily.

Ad -2-
Several options:
- during our main build, the build pulls (tx pull) from transifex. Is
what we do now: Transifex is main repo for translations. IF an
individual wants to build a translated doc, she/he needs to pull the
translations from transifex itself (which is a disadvantage, as pulling
translations from tx is SLOW...)
- we run a daily script which pulls the translations from tx and commits
those po files (for ALL languages to be build) in git-repo.
Main advantage: we keep being boss over the po files ourselves, AND
everybody can build translated docs, because the po files are (well 1
day old...) in git
- use the Transifex Github integration (I played with it one year ago,
and then it could not handle our long resource handles at that time...)

I think I prefer the second or third option, will push nl, it and fr
translations into git later today (by just pulling the 3.4 translations
from build server), so everybody can try to build translated sites too.
Als together with maybe building it on travis, it is probably the best
way..

Want to see current status?

- vanilla Sphinx local ( EVERYBODY can try to build branch qgisdoc2020
NOW on windows and linux, see README [0] )

- vanilla Sphinx remote ('make site' works if you can like me rsync to
qgis2 [1]) (NOTE: a lot has to be done there)

- everybody should now be able to 'just edit' on github [2]

- 'make doctest' (== building in te qgis/qgis docker image) is also
working, see travis [4] qgisdoc2020 branch there


Sorry for being rather long, and while writing maybe answering my own
questions :-)

But I really really hope we can keep stuff as simple as possible and
encourage everybody to JOIN. Please contact us via documentation list

About PDF's: maybe later... but I'm OK without those...
So if you want them: join us

Regards,

Richard Duivenvoorde


[0] https://github.com/qgis/QGIS-Documentation/tree/qgisdoc2020
[1] https://docs.qgis.org/3.10/
[3]
https://github.com/qgis/QGIS-Documentation/blob/qgisdoc2020/docs/user_manual/introduction/getting_started.rst
[4] https://travis-ci.org/github/qgis/QGIS-Documentation
[5] https://docs.transifex.com/transifex-github-integrations/github-tx-ui
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Doc Travis

[matteo]

Thanks for all the hints. Apparently we were missing something silly
(that's the effect of a remote HackFest :D )

anyway, travis is now happy with both documentation and pyqgis test
(thanks Richard for the fix).

I really think this is the perfect moment to move on to the rtd theme
and release the 3.10 documentation. So what are the blockers? can we
write a todo list and squash all the problems?

That's what comes into my mind:

* verify Transifex pulling/pushing
* apache on the server to publish the docs
* ... what else?

Cheers

Matteo
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer