Re: [Qgis-developer] Processing algs & plugins... should we actively pull into master?

[Ari Jolma]

Interesting. Let me bring my relative outsider's and newcomer's view 
into this. Although a long time QGIS user, I've never really dug into 
the Processing tools.


I'm working on a project, where one part is a plugin - mostly a 
relatively simple set of dialogs that integrate with other parts of the 
QGIS GUI. In another part of the project I develop datasets - and I've 
written some processing tools for that based on GDAL. I just yesterday 
cleaned two of them up a bit and put them into github and mentioned that 
on gdal-dev (I cc this email to gdal-dev). They're Perl programs that 
use also other Perl modules.


My intent is to release the plugin to my end users through a website - 
the QGIS plugin system allows one to add new library URLs, so I just 
need to write an XML file and put the plugin into a zip and upload that 
to our server to release a new version. I can make that into "make dist" 
or something. I'd like to keep this separate from the official plugin 
repository for separate identity and maybe because the plugin is really 
for this specific purpose without wider interest at least for now.


I just studied the Processing tools because it seems that maybe I could 
also make my processing tools available for my end users (or at least as 
a formal deliverable of our project) that way. There seems to be two 
possible approaches: 1) get the tools into QGIS-Processing or 2) make a 
distribution and ask the end users to install them and use the "Add 
script from file" tool in Processing tools (won't work out of box 
because python extension is hard coded there). The tools I've made are 
pretty generic but I wrote them as pure CLI tools and the other one I 
use for huge rasters so it takes a lot of time why I put them running 
into some cloud computers. Anyway I definitely see a benefit in running 
them from within QGIS and distributing them to end users - and with an 
additional code layer the input and output could be tied to layers in 
QGIS and dialog boxes. The same way as GDAL utilities are now usable 
from QGIS.


The QGIS Processing tools is already now a huge collection of tools. 
Some tools are there and directly usable (GDAL for example since I have 
GDAL in my computer), some tools can be downloaded for use, and some 
tools require that I install more software into my computer (TauDEM for 
example).


I see dangers in that one can easily download code from the web into 
his/her computer and execute it easily. This will be an issue at least 
with IT support of my end users.


I see the benefit of peer control/help/review over add-ons I develop and 
make usable by others in QGIS.


Right now I'm a bit confused and also perhaps awed by the potential of 
the QGIS Processing tools.


Cheers,

Ari


08.02.2017, 18:49, Alex M kirjoitti:

On 02/06/2017 01:24 AM, Paolo Cavallini wrote:

Il 06/02/2017 10:21, Victor Olaya ha scritto:


Just let make sure that we have some clear acceptance criteria (algs
with tests, trying to follow some "best practices" and using
Processing methods when possible, etc).

right, what could be the procedure then? whan I see a submitted pluygin
that qualifies for this should I warn you (and Nyall) for analysis
before suggesting it to the dev?
All the best.


I thought that https://github.com/qgis/QGIS-Processing was intended for
these kinds of algorithms. It's centralized, but not core, intended for
pure methods instead of plugins.

Perhaps the confusion is that the Plugins and the Processing add-ons
have 2 different installers, and it's not obvious that both exist?

This seems like the start down the path, where after addition of tests
and code review it might then go to core.

Thanks,
Alex
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer


___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Processing algs & plugins... should we actively pull into master?

[Alex M]

On 02/06/2017 01:24 AM, Paolo Cavallini wrote:
> Il 06/02/2017 10:21, Victor Olaya ha scritto:
> 
>> Just let make sure that we have some clear acceptance criteria (algs
>> with tests, trying to follow some "best practices" and using
>> Processing methods when possible, etc).
> 
> right, what could be the procedure then? whan I see a submitted pluygin
> that qualifies for this should I warn you (and Nyall) for analysis
> before suggesting it to the dev?
> All the best.
> 

I thought that https://github.com/qgis/QGIS-Processing was intended for
these kinds of algorithms. It's centralized, but not core, intended for
pure methods instead of plugins.

Perhaps the confusion is that the Plugins and the Processing add-ons
have 2 different installers, and it's not obvious that both exist?

This seems like the start down the path, where after addition of tests
and code review it might then go to core.

Thanks,
Alex
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Processing algs & plugins... should we actively pull into master?

[Tim Sutton]

Hi

> On 06 Feb 2017, at 12:35 PM, Victor Olaya  wrote:
> 
> 
> One thing I am concerned about is that we guarantee some stability in the 
> behavior of processing. While it is true that strictly speaking new 
> algorithms do not represent API extensions / additions, in some level I think 
> they do since anyone using Python + Processing will be vulnerable to future 
> behavioral changes / deletions from the set of tools shipped with processing. 
> If we want to get plugin writers to rely on Processing rather than 
> implementing geoprocessing etc.  functionality themselves, they need to know 
> that a) the algorithms they need will be installed on their user's computers 
> and b) that those algorithms will behave in a consistent way across the major 
> release life cycle of QGIS.
> 
> 
> I think we can have good control over that. Once the code is in the QGIS repo 
> and authors work on it there via PRs, we shouldnt allow them to modify 
> algorithm semantics or do any 'crazy' change. We have control over that. So 
> actually it is better to have it in core for that kind of integration, since 
> we, as QGIS project , will likely be more strict than a developer working on 
> his own repo, and take all this into account.

Yeah that makes sense  

Thanks

Tim

> 

—










Tim Sutton

Co-founder: Kartoza
Project chair: QGIS.org

Visit http://kartoza.com  to find out about open source:

Desktop GIS programming services
Geospatial web development
GIS Training
Consulting Services

Skype: timlinux 
IRC: timlinux on #qgis at freenode.net

Kartoza is a merger between Linfiniti and Afrispatial

___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Processing algs & plugins... should we actively pull into master?

[Victor Olaya]

>
>
> One thing I am concerned about is that we guarantee some stability in the
> behavior of processing. While it is true that strictly speaking new
> algorithms do not represent API extensions / additions, in some level I
> think they do since anyone using Python + Processing will be vulnerable to
> future behavioral changes / deletions from the set of tools shipped with
> processing. If we want to get plugin writers to rely on Processing rather
> than implementing geoprocessing etc.  functionality themselves, they need
> to know that a) the algorithms they need will be installed on their user's
> computers and b) that those algorithms will behave in a consistent way
> across the major release life cycle of QGIS.
>
>
I think we can have good control over that. Once the code is in the QGIS
repo and authors work on it there via PRs, we shouldnt allow them to modify
algorithm semantics or do any 'crazy' change. We have control over that. So
actually it is better to have it in core for that kind of integration,
since we, as QGIS project , will likely be more strict than a developer
working on his own repo, and take all this into account.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Processing algs & plugins... should we actively pull into master?

[Nyall Dawson]

On 6 February 2017 at 19:21, Victor Olaya  wrote:
> Very interesting question...
>
> IMHO, all plugins that include processing algorithms like that (just
> one of them or a few of them, using only native code), we should ask
> developers to put the code in the central QGIS repo, and then work on
> that through PRs. It makes more sense. In the case of a plugin that
> adds a large group of algorithms and has some "identity", or uses an
> external app, I think it makes sense to have it as a separate plugin.
>

Ah yes - I forgot to include that in my email, even though that was my
intent. Definitely plugins with external deps (or even large amounts
of private classes) should remain as plugins.

> Just let make sure that we have some clear acceptance criteria (algs
> with tests, trying to follow some "best practices" and using
> Processing methods when possible, etc).

Big +1. We'd still need to careful vet and review the code for quality.

Nyall
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Processing algs & plugins... should we actively pull into master?

[Tim Sutton]

Hi

> On 06 Feb 2017, at 11:21 AM, Victor Olaya  wrote:
> 
> Very interesting question...
> 
> IMHO, all plugins that include processing algorithms like that (just
> one of them or a few of them, using only native code), we should ask
> developers to put the code in the central QGIS repo, and then work on
> that through PRs. It makes more sense. In the case of a plugin that
> adds a large group of algorithms and has some "identity", or uses an
> external app, I think it makes sense to have it as a separate plugin.

Or even just a discrete group in the Processing toolbox?

> 
> I think most developers will agree on that and will accept it. If
> someone doesntwell, then we have that code as a separate plugin,
> not that bad, but with those that accept, we will enlarge the core
> Processing collection and make it better.
> 
> Just let make sure that we have some clear acceptance criteria (algs
> with tests, trying to follow some "best practices" and using
> Processing methods when possible, etc).
> 
> I will be happy to help on that, even helping devs in this migration
> from separate plugin to core Processing, in case there is a need to
> convert code somehow
> 

I know there is a bit of philosophical thinking to be done as to whether we 
want to be a 'slim core' project or a 'batteries included' project. I generally 
prefer the batteries included approach if we can make features discoverable and 
consistent with the rest of the QGIS experience. So +1 from me to build out 
processing into a much richer store of tools.  One more argument for including 
new algs in processing in QGIS Master rather than as plugins - if we want 
plugin authors to utilize processing more, it is much more convenient that the 
algs are all there 'out of the box' compared to expecting plugin users to 
install other plugins in order to first satisfy processing dependencies.

One thing I am concerned about is that we guarantee some stability in the 
behavior of processing. While it is true that strictly speaking new algorithms 
do not represent API extensions / additions, in some level I think they do 
since anyone using Python + Processing will be vulnerable to future behavioral 
changes / deletions from the set of tools shipped with processing. If we want 
to get plugin writers to rely on Processing rather than implementing 
geoprocessing etc.  functionality themselves, they need to know that a) the 
algorithms they need will be installed on their user's computers and b) that 
those algorithms will behave in a consistent way across the major release life 
cycle of QGIS.

Regards

Tim


> Cheers
> 
> 
> 2017-02-06 9:46 GMT+01:00 Paolo Cavallini :
>> Il 06/02/2017 00:55, Nyall Dawson ha scritto:
>> 
>>> What I'm wondering now is whether we should be merging functionality
>>> like this into master.
>> 
>> +1, thanks for raising this.
>> 
>>> - might be a good spring-board for easing plugin developers into
>>> working with the master repo. There's a chance they'll find confidence
>>> in this to make changes elsewhere in the code.
>> 
>> exactly, I'm always trying to convince plugin developers to jump in the
>> bandwagon, this move would be a strong argument for it.
>> 
>>> fixes/changes, and need to wait for next QGIS release for the changes
>>> to be distributed
>> 
>> a general issue with Processing, and other core plugins BTW.
>> better discuss about this in general terms.
>> 
>>> - we risk "stepping on plugin developer's toes". It's possible a
>>> plugin developer might not like seeing the role their plugin once
>>> played "taken over" by master. (On the other hand, they may be proud
>>> to see their work accepted into the default install!)
>> 
>> this is a possibility; in my experience however it can be avoided by
>> proper communication with plugin authors. for what I have seen, plugin
>> authors are pleased by the interest we have in their work, so I think
>> we'll encounter no major problems here. I can keep on taking the
>> responsibility of communicating to plugin authors if nobody wants to
>> step in.
>> 
>>> So what do we think? Good move? Bad move?
>> 
>> good by all means.
>> All the best.
>> 
>> --
>> Paolo Cavallini - www.faunalia.eu
>> QGIS & PostGIS courses: http://www.faunalia.eu/training.html
>> https://www.google.com/trends/explore?date=all&geo=IT&q=qgis,arcgis
>> ___
>> Qgis-developer mailing list
>> Qgis-developer@lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

—










Tim Sutton

Co-founder: Kartoza
Project chair: QGIS.org

Visit http://kartoza.com  to find out about open source:

Des

Re: [Qgis-developer] Processing algs & plugins... should we actively pull into master?

[Paolo Cavallini]

Il 06/02/2017 10:21, Victor Olaya ha scritto:

> Just let make sure that we have some clear acceptance criteria (algs
> with tests, trying to follow some "best practices" and using
> Processing methods when possible, etc).

right, what could be the procedure then? whan I see a submitted pluygin
that qualifies for this should I warn you (and Nyall) for analysis
before suggesting it to the dev?
All the best.

-- 
Paolo Cavallini - www.faunalia.eu
QGIS & PostGIS courses: http://www.faunalia.eu/training.html
https://www.google.com/trends/explore?date=all&geo=IT&q=qgis,arcgis
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Processing algs & plugins... should we actively pull into master?

[Victor Olaya]

Very interesting question...

IMHO, all plugins that include processing algorithms like that (just
one of them or a few of them, using only native code), we should ask
developers to put the code in the central QGIS repo, and then work on
that through PRs. It makes more sense. In the case of a plugin that
adds a large group of algorithms and has some "identity", or uses an
external app, I think it makes sense to have it as a separate plugin.

I think most developers will agree on that and will accept it. If
someone doesntwell, then we have that code as a separate plugin,
not that bad, but with those that accept, we will enlarge the core
Processing collection and make it better.

Just let make sure that we have some clear acceptance criteria (algs
with tests, trying to follow some "best practices" and using
Processing methods when possible, etc).

I will be happy to help on that, even helping devs in this migration
from separate plugin to core Processing, in case there is a need to
convert code somehow

Cheers


2017-02-06 9:46 GMT+01:00 Paolo Cavallini :
> Il 06/02/2017 00:55, Nyall Dawson ha scritto:
>
>> What I'm wondering now is whether we should be merging functionality
>> like this into master.
>
> +1, thanks for raising this.
>
>> - might be a good spring-board for easing plugin developers into
>> working with the master repo. There's a chance they'll find confidence
>> in this to make changes elsewhere in the code.
>
> exactly, I'm always trying to convince plugin developers to jump in the
> bandwagon, this move would be a strong argument for it.
>
>> fixes/changes, and need to wait for next QGIS release for the changes
>> to be distributed
>
> a general issue with Processing, and other core plugins BTW.
> better discuss about this in general terms.
>
>> - we risk "stepping on plugin developer's toes". It's possible a
>> plugin developer might not like seeing the role their plugin once
>> played "taken over" by master. (On the other hand, they may be proud
>> to see their work accepted into the default install!)
>
> this is a possibility; in my experience however it can be avoided by
> proper communication with plugin authors. for what I have seen, plugin
> authors are pleased by the interest we have in their work, so I think
> we'll encounter no major problems here. I can keep on taking the
> responsibility of communicating to plugin authors if nobody wants to
> step in.
>
>> So what do we think? Good move? Bad move?
>
> good by all means.
> All the best.
>
> --
> Paolo Cavallini - www.faunalia.eu
> QGIS & PostGIS courses: http://www.faunalia.eu/training.html
> https://www.google.com/trends/explore?date=all&geo=IT&q=qgis,arcgis
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Processing algs & plugins... should we actively pull into master?

[Paolo Cavallini]

Il 06/02/2017 00:55, Nyall Dawson ha scritto:

> What I'm wondering now is whether we should be merging functionality
> like this into master.

+1, thanks for raising this.

> - might be a good spring-board for easing plugin developers into
> working with the master repo. There's a chance they'll find confidence
> in this to make changes elsewhere in the code.

exactly, I'm always trying to convince plugin developers to jump in the
bandwagon, this move would be a strong argument for it.

> fixes/changes, and need to wait for next QGIS release for the changes
> to be distributed

a general issue with Processing, and other core plugins BTW.
better discuss about this in general terms.

> - we risk "stepping on plugin developer's toes". It's possible a
> plugin developer might not like seeing the role their plugin once
> played "taken over" by master. (On the other hand, they may be proud
> to see their work accepted into the default install!)

this is a possibility; in my experience however it can be avoided by
proper communication with plugin authors. for what I have seen, plugin
authors are pleased by the interest we have in their work, so I think
we'll encounter no major problems here. I can keep on taking the
responsibility of communicating to plugin authors if nobody wants to
step in.

> So what do we think? Good move? Bad move?

good by all means.
All the best.

-- 
Paolo Cavallini - www.faunalia.eu
QGIS & PostGIS courses: http://www.faunalia.eu/training.html
https://www.google.com/trends/explore?date=all&geo=IT&q=qgis,arcgis
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer