Thanks Vaclav for your detailed analysis. As expected, it seems that there are a number of common tasks between GRASS and OTB, possibly also SAGA, so it make sense to keep a joint discussion. Please consider participating even remotely to the HackFest discussion: https://github.com/qgis/QGIS/wiki/DeveloperMeetingMadeira2018#future-of-processing-providers All the best.
Il 09/02/2018 21:33, Vaclav Petras ha scritto: > Dear Nyall and all, > > here are some of my thoughts mostly relating to GRASS GIS (although it > may not express views of the whole GRASS GIS developer team). > > On Thu, Feb 8, 2018 at 8:55 PM, Nyall Dawson <nyall.daw...@gmail.com > <mailto:nyall.daw...@gmail.com>> wrote: >> >> Here's the situation as I see it: >> >> ------------ >> The past >> ----------- >> >> We (the QGIS project) have struggled to keep a bunch of providers >> stable and available with the base QGIS install, and the current >> maintainers (Alex and Victor, others) have (understandably) struggled >> with the burden of maintaining these alone and the time commitment >> required to do so. Users have been frustrated by breakage which occurs >> when the algorithms they depend on break due to changes in underlying >> libraries for which the processing providers have not been adapted. >> >> Despite this, I'd say overall it's worked OK-ish up to now, but >> definitely with lots of room for improvement. >> >> ------------- >> The goals >> ------------- >> >> I think everyone involved is in agreement that processing's strength >> comes when there's many providers available and it's able to tie >> together processes regardless of which provider or library they come >> from. So I'd say our common goals are: >> >> 1. Encourage development of as many processing providers as possible >> 2. Make it easy for developers to create and maintain these providers >> 3. Make it easy for users to be able to use the providers >> 4. Make everything stable and painfree for users >> >> Any disagreements? No? Good ;) >> >> So if we agree that those are the end goals, we should be using them >> to drive this discussion. >> >> ------------------- >> The questions >> ------------------- >> >> The open, debatable questions are: >> >> - Which is the best approach for allowing easy maintenance of >> providers (*regardless* of whether the provider is maintained by the >> core QGIS team or a 3rd party)? Is it keeping the code in the master >> codebase and locking to QGIS releases, or publishing via plugins and >> having independent release schedules? Which approach will encourage >> more active maintenance of these providers and share the burden of >> this maintenance? > > One thing is where to keep the code, however I'm not sure what code are > we talking about and I would like to talk about this first. I think that > recent post by Moritz is bringing this up as well: > > https://lists.osgeo.org/pipermail/grass-dev/2018-February/087420.html > <https://lists.osgeo.org/pipermail/grass-dev/2018-February/087420.html> > > As far as I know, QGIS Processing has a text file for each GRASS module > describing its interface and maintenance of these takes time. However, > every GRASS module can tell about itself when called with > --interface-description parameter. If this is used, individual > parameters don't need to be maintained and any, even user provided > module can be executed in processing. > > The --interface-description parameter gives XML which would need to be > converted or a new parameter, let's say --qgis-description, would need > to be added for a QGIS-specific format, there is for example --script > for GRASS GIS interface description for scripts. --qgis-description > would need to be in GRASS GIS code base while the conversion can be > anywhere. See emails from Rashad discussing a possible implementation > with the --qgis-description way and the OTB way: > > https://lists.osgeo.org/pipermail/grass-dev/2018-February/087362.html > <https://lists.osgeo.org/pipermail/grass-dev/2018-February/087362.html> > https://lists.osgeo.org/pipermail/grass-dev/2018-February/087390.html > <https://lists.osgeo.org/pipermail/grass-dev/2018-February/087390.html> > > The was discussed in the past and in fact, it is used in the QGIS GRASS > plugin as Radim explains in this older email: > > https://lists.osgeo.org/pipermail/grass-dev/2015-March/074629.html > <https://lists.osgeo.org/pipermail/grass-dev/2015-March/074629.html> > > However, it is not using the --interface-description XML only, but it is > also using the qgm files to supply some additional information which > means that this system still requires updates which are separate from > the updates of GRASS modules. This can be avoided if the necessary > information is updated upstream or sometimes even GRASS > --interface-description format or the individual module descriptions are > extended to include that what is needed by QGIS Processing. Here is a > thread which disuses this issues although diverges into the following: > > https://lists.osgeo.org/pipermail/grass-dev/2015-September/076138.html > <https://lists.osgeo.org/pipermail/grass-dev/2015-September/076138.html> > > One issue which is creating differences between QGIS Processing > representation of the module and the GRASS one is the issue of advanced > parameters. GRASS itself has mechanism to organize the parameters into > groups and marks the required ones. This is of course something which is > constantly being refined and it can be used and changed also for QGIS > Processing as discussed in this 2015 post: > > https://lists.osgeo.org/pipermail/grass-dev/2015-December/078000.html > <https://lists.osgeo.org/pipermail/grass-dev/2015-December/078000.html> > > Other issues besides the interface include list of GRASS modules usable > and unusable in QGIS Processing, thematic tree of these modules, > splitting modules and more. Many of these are discussed in this recent post: > > https://lists.osgeo.org/pipermail/grass-dev/2018-February/087388.html > <https://lists.osgeo.org/pipermail/grass-dev/2018-February/087388.html> > > One of the issues is issue of formats (or import & export). Here again, > it seems to that it is more advantageous to have a function in GRASS > code base to care take of the import and, if needed, push the changes > upstream to it (from QGIS point of view). Here is a related example > which is a comment from Markus Neteler (regarding currently preferred > solution for vector import) which would be part of PR/patch review > process if the import function was originally submitted as extension of > GRASS code base: > > https://github.com/qgis/QGIS/pull/5426#issuecomment-345067457 > <https://github.com/qgis/QGIS/pull/5426#issuecomment-345067457> > > Then there is of course the issue of GRASS modules which are more > difficult to wrap in QGIS Processing because the inputs and outputs are > not explicitly stated, most prominent example is probably r.mapcalc but > there is couple of more, although we were able to reduce this number lately: > > https://lists.osgeo.org/pipermail/grass-dev/2015-January/072979.html > <https://lists.osgeo.org/pipermail/grass-dev/2015-January/072979.html> > > See also the following post from Giovanni Manghi (sent off-list > to Markus Neteler and shared by him) and a ticket commend by me for more > examples and details: > > https://lists.osgeo.org/pipermail/grass-dev/2014-August/070342.html > <https://lists.osgeo.org/pipermail/grass-dev/2014-August/070342.html> > https://trac.osgeo.org/grass/ticket/2409#comment:17 > <https://trac.osgeo.org/grass/ticket/2409#comment:17> > > Here is a good r.mapcalc-related example of (at least) asking for > changes upstream (although sent privately to Markus Neteler who opened > the issue): > > https://trac.osgeo.org/grass/ticket/3431 > <https://trac.osgeo.org/grass/ticket/3431> > >> - Who should be responsible for maintaining the providers? Should >> maintenance be done by the QGIS core developer team or by 3rd parties? > > I'm not sure if we can really separate this from the place in the code, > but if there is a notion of pushing changes upstream then that gives > definitively us more flexibility. > > Besides maintenance, I think the initial development, which is necessary > for a realistic maintenance plan, is a major issue. I, and I think some > other GRASS devs too, would be happy to work on it if the financing is > addressed. A different suggestion is here as well and that's GSoC idea > proposed by Martin Landa: > > https://trac.osgeo.org/grass/wiki/GSoC/2018#ImproveGRASSintegrationinQGIS3 > <https://trac.osgeo.org/grass/wiki/GSoC/2018#ImproveGRASSintegrationinQGIS3> > >> - Is it a core requirement that 3rd party providers are installed with >> EVERY QGIS install, or is it acceptable to require interested users to >> manually install the tools which they find useful? If the second >> option is preferred, then how can we ensure that users are aware of >> the availability of these tools? > > If they install manually, on Windows, users can have the latest version > of the software instead of the one packaged with QGIS. OSGeo4W is a > different case; not sure about Mac. However, on Linux, the decision is > up to the packagers, so in any case, the QGIS part needs to be able to > work with different versions. Regardless of QGIS packaging efforts, I > have seen many users with old GRASS who installed new standalone GRASS > on the side, even recently. > > On the other hand, many users benefit from the one package, because they > need to get software approved, if they get QGIS packaged with GRASS > approve, they got GRASS as well. Getting approval for each individual > provider is likely more paperwork. > > I hope this will help and let me know what I'm missing. > > Best, > Vaclav > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > > _______________________________________________ > saga-gis-developer mailing list > saga-gis-develo...@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/saga-gis-developer > -- Paolo Cavallini - www.faunalia.eu QGIS & PostGIS courses: http://www.faunalia.eu/training.html https://www.google.com/trends/explore?date=all&geo=IT&q=qgis,arcgis _______________________________________________ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev