Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
On Wed, Apr 23, 2014 at 4:02 PM, Sören Gebbert Vector lib does not call G_fatal_error(), it was written in parallel with QGIS plugin with vector editing in mind. I hope it is still true. I fear that might be not the case anymore. For example, all memory allocation functions G_calloc(), G_malloc() and G_realloc() will call G_fatal_error() in case the memory allocation fails. As far as i can tell the vector library make often use of this functions. So we have the problem that basic core functions call G_fatal_error() that are used all over the place in GRASS GIS. I don't care about allocation error. If it really runs out of memory then exit() is maybe the best. Any other action (saving project, saving changes) will also need memory and thus fail. Are you often running out of memory? It made me curious how Qt handles allocation problem, because I think that allocation results are not checked in the whole QGIS at all. For example QString::realloc(int) on the pointer returned by malloc() calls Q_CHECK_PTR() macro which prints error if pointer is 0. Then it continues execution even if allocation failed. If you patch GRASS to use setjmp/longjmp to implement an exception like behavior then you need to write clean up code for static variables and dynamically allocated memory. A G_fatal_error() call can be deeply nested in a library function. Cleaning only the static variables to be in usable state will lead to problematic memory leaks. So it is a complex task in GRASS. I don't want to patch GRASS until it is really inevitable. The next one is my choice: Using setjmp/longjmp in QGIS wrapping all GRASS functions that might call G_fatal_error() is complex, since a G_fatal_error() call can be deeply nested in a library function. You need to keep track of this resulting in plenty setjmp wrappers around GRASS functions. setjmp will be called just before the top level function we need. You need to cleanup global static structures as well and you will have no control over pointers pointing to dynamic allocated memory that got lost in a longjmp. and will not prevent QGIS from terminating or segfaulting eventually in case of a fatal error. Why? Because of the undefined state of global variables and structures and because of potential memory leaks that may sum up making QGIS unusable. In theory maybe, in practice it will work and it DOES work with GRASS 6. I don't care about few bytes lost. G_fatal_error() is really rare in QGIS provider/plugin, it should be rare. Radim ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
Sören Gebbert wrote: Vector lib does not call G_fatal_error(), it was written in parallel with QGIS plugin with vector editing in mind. I hope it is still true. I fear that might be not the case anymore. For example, all memory allocation functions G_calloc(), G_malloc() and G_realloc() will call G_fatal_error() in case the memory allocation fails. As far as i can tell the vector library make often use of this functions. So we have the problem that basic core functions call G_fatal_error() that are used all over the place in GRASS GIS. Memory allocation is largely a theoretical problem. If you're out of memory, there's no guarantee that whatever error-handling mechanism you come up with can function (e.g. printing an error message may result in a malloc() within libc's stdio functions). If you patch GRASS to use setjmp/longjmp to implement an exception like behavior then you need to write clean up code for static variables and dynamically allocated memory. A G_fatal_error() call can be deeply nested in a library function. Cleaning only the static variables to be in usable state will lead to problematic memory leaks. So it is a complex task in GRASS. Memory leaks are another area where GRASS libraries aren't particularly suitable for persistent applications. In general, any leak which has no significance for a command-line module is considered not significant, period. Many strings inside the library are dynamically allocated and never freed (rather, they're implicitly freed upon process termination). This means that the library can just return a pointer and nothing needs to worry about that pointer becoming invalid (because it will remain valid for the lifetime of the process). Allowing such data to be freed would require either keeping track of usage (e.g. reference counting) or duplicating the underlying data rather than simply returning a pointer to it. Using setjmp/longjmp in QGIS wrapping all GRASS functions that might call G_fatal_error() is complex, since a G_fatal_error() call can be deeply nested in a library function. You need to keep track of this resulting in plenty setjmp wrappers around GRASS functions. You need to cleanup global static structures as well and you will have no control over pointers pointing to dynamic allocated memory that got lost in a longjmp. If it can be gotten to the point where memory leaks are the only problem, there isn't really that much of a problem, given that we're talking about the behaviour in exceptional conditions. The more fundamental problem is e.g. allocating a slot from an array, starting to fill in its fields, and aborting (longjmp, exception, etc) part-way through. Or doing something similar with groups of related variables. Anything which iterates over the array is going to encounter a slot where some of the fields contain garbage. If those fields are pointers, offsets, lengths, etc, the likely result is a segfault. Having a kind of garbage collector in GRASS, implemented in the G_*alloc() functions, may solve many problems. But i have no idea howto implement the garbage collector to detect pointers that got orphaned in a longjmp to automagically clean them. I don't think that's realistic. There are simpler things which would help in many cases; e.g. in the slots case, not allocating the slot until after you've obtained all of the values which are needed to initialise it. Beyond that, we have to deal with C's lack of destructors. We may want to maintain a stack of cleanup handlers (i.e. destructors) which would be executed in the event of a fatal error. Code which needs to ensure some form of cleanup occurs would push handlers as resources are allocated, and pop them prior to a normal return. This would avoid having to implement nested try/catch constructs using setjmp/longjmp and G_set_error_routine. In fact, that currently isn't possible, as you can't get at the existing fatal error handler in order to save and restore it around an local handler. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
2014-04-20 11:38 GMT+02:00 Martin Landa landa.mar...@gmail.com: Hi, 2014-04-20 1:34 GMT+02:00 Sören Gebbert soerengebb...@googlemail.com: Hence the wrapper will not only do simple function wrapping, it will also implement a higher level interface: [...] what about to create a trac wiki page dedicated to RFC and collect this notes there... ? https://trac.osgeo.org/grass/wiki/Grass7/RPCInterface Martin -- Martin Landa * http://geo.fsv.cvut.cz/gwiki/Landa ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
On Tue, Apr 22, 2014 at 4:49 PM, Sören Gebbert soerengebb...@googlemail.com wrote: IMHO we GRASS developers are too stubborn to change the design of GRASS GIS to work as library with persistent applications. This would require rewriting plenty of core functionality and modification of all C-modules. But there is a solution for persistent applications, called Remote Procedure Call (RPC), more below. It sounds interesting but it seems to be a huge work an far future. I Its far less work then modifying GRASS libraries to run with persistent applications. We have already the providers and the plugin working with GRASS 6, I just want to upgrade it to GRASS 7. The only problem I found so far is that G__get_window() and Rast_get_cellhd() call G_fatal_error(). I can only dedicate few hours/max days of my free time to that, so I have to consider well what is less work, RPC is not. We need such an approach anyway, since vector editing is a GUI task - GUI is a persistent application - we need an exit safe interface to the GRASS libraries to implement a reliable vector editing GUI in GRASS itself. Vector lib does not call G_fatal_error(), it was written in parallel with QGIS plugin with vector editing in mind. I hope it is still true. am also concerned about complexity, e.g. starting / keeping running servers on various platforms. I can imagine endless bug reports about server not running. I am looking for something which will not Since only QGIS is spawning the server process, it is always able to check if the process is running and can restart it when needed. We already make use of this concept in GRASS and it works nicely[1]. Spawning of processes and keeping track of them is an easy task with Qt, and it is completely OS independent. The Qt Bitcoin client for example makes use of RPC and runs on Windows, Mac OS and various Linux systems. So i don't see a problem here. The raster provider is already using something like that. Raster data are read from executed GRASS module output (a module written for that in QGIS in fact) and identify tool keeps another GRASS module running and it is communicating with that through a pipe. I wrote that as an experiment and because I know, how much unnecessary additional code and potential problems it is, I don't want to continue in that direction. require permanent assistance. It is not KISS enough for me. What would be a better KISS solution? IMO the RPC approach will make the implementation of the QGIS-GRASS data provider and the plugin simpler. There will be no mixing of C++ and C-code anymore. We are no starting the plugin from scratch, it is already written, we just need to make it work with GRASS 7. No setjmp/longjmp/exception workaround that is really complex Why? and will not prevent QGIS from terminating or segfaulting eventually in case of a fatal error. Why? Radim The QGIS GRASS data provider/plugin will be a clean C++ implementation using the thrift library. It will be better maintainable since developers do not need to know all the different GRASS libraries (gis, raster, vector, database) but only the RPC interface that make use of a subset of the GRASS library functions, bundling them in an object oriented API. QGIS do not need to link against GRASS libraries anymore, future GRASS versions will be supported out of the box. This is much more KISS then the current approach. Best regards Soeren [1] http://grass.osgeo.org/programming7/namespacepython_1_1temporal_1_1c__libraries__interface.html Radim 2014-04-18 11:31 GMT+02:00 Radim Blazek radim.bla...@gmail.com: I have upgraded the vector provider to GRASS 7, layers may be added by drag from browser. The raster and the plugin are disabled. Be careful about multiple versions on the same system (LD_LIBRARY_PATH..., check with ldd if does not work). Unfortunately GRASS 7 moved ahead towards its aim to make life harder for anyone trying to use the GRASS libraries [1]. Basically more and more low level functions are calling exit() instead of returning error code if something failed. As I am not willing to implement GRASS module call for each simple function, we have to think again about hacks we can use: 1) add a requirement that GRASS 7 used with QGIS must be compiled with -fexceptions 2) add a requirement that a patch (it is a single line comment in fact) must be applied to GRASS 7 to make it usable with QGIS 3) use setjmp()/longjmp() 4) let QGIS crash whenever GRASS lib function fails I fear that none of these suggestions are good working solutions, since in case a fatal error or a segfault occurs nothing (no exception, no setjmp) will prevent QGIS from crashing. So i would like to suggest the following: 5.) Using a RPC interface for map metadata, vector and raster map access. Hence one or several GRASS server processes provide an RPC interface to access GRASS library functions that are needed in a persistent
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
Hi Radim, 2014-04-23 14:50 GMT+02:00 Radim Blazek radim.bla...@gmail.com: On Tue, Apr 22, 2014 at 4:49 PM, Sören Gebbert soerengebb...@googlemail.com wrote: IMHO we GRASS developers are too stubborn to change the design of GRASS GIS to work as library with persistent applications. This would require rewriting plenty of core functionality and modification of all C-modules. But there is a solution for persistent applications, called Remote Procedure Call (RPC), more below. It sounds interesting but it seems to be a huge work an far future. I Its far less work then modifying GRASS libraries to run with persistent applications. We have already the providers and the plugin working with GRASS 6, I just want to upgrade it to GRASS 7. The only problem I found so far is that G__get_window() and Rast_get_cellhd() call G_fatal_error(). I can only dedicate few hours/max days of my free time to that, so I have to consider well what is less work, RPC is not. Yes, i absolutely understand this. We need such an approach anyway, since vector editing is a GUI task - GUI is a persistent application - we need an exit safe interface to the GRASS libraries to implement a reliable vector editing GUI in GRASS itself. Vector lib does not call G_fatal_error(), it was written in parallel with QGIS plugin with vector editing in mind. I hope it is still true. I fear that might be not the case anymore. For example, all memory allocation functions G_calloc(), G_malloc() and G_realloc() will call G_fatal_error() in case the memory allocation fails. As far as i can tell the vector library make often use of this functions. So we have the problem that basic core functions call G_fatal_error() that are used all over the place in GRASS GIS. am also concerned about complexity, e.g. starting / keeping running servers on various platforms. I can imagine endless bug reports about server not running. I am looking for something which will not Since only QGIS is spawning the server process, it is always able to check if the process is running and can restart it when needed. We already make use of this concept in GRASS and it works nicely[1]. Spawning of processes and keeping track of them is an easy task with Qt, and it is completely OS independent. The Qt Bitcoin client for example makes use of RPC and runs on Windows, Mac OS and various Linux systems. So i don't see a problem here. The raster provider is already using something like that. Raster data are read from executed GRASS module output (a module written for that in QGIS in fact) and identify tool keeps another GRASS module running and it is communicating with that through a pipe. I wrote that as an experiment and because I know, how much unnecessary additional code and potential problems it is, I don't want to continue in that direction. Ok. require permanent assistance. It is not KISS enough for me. What would be a better KISS solution? IMO the RPC approach will make the implementation of the QGIS-GRASS data provider and the plugin simpler. There will be no mixing of C++ and C-code anymore. We are no starting the plugin from scratch, it is already written, we just need to make it work with GRASS 7. No setjmp/longjmp/exception workaround that is really complex Why? If you patch GRASS to use setjmp/longjmp to implement an exception like behavior then you need to write clean up code for static variables and dynamically allocated memory. A G_fatal_error() call can be deeply nested in a library function. Cleaning only the static variables to be in usable state will lead to problematic memory leaks. So it is a complex task in GRASS. Using setjmp/longjmp in QGIS wrapping all GRASS functions that might call G_fatal_error() is complex, since a G_fatal_error() call can be deeply nested in a library function. You need to keep track of this resulting in plenty setjmp wrappers around GRASS functions. You need to cleanup global static structures as well and you will have no control over pointers pointing to dynamic allocated memory that got lost in a longjmp. IMHO booth cases require plenty of work to make them usable. Having a kind of garbage collector in GRASS, implemented in the G_*alloc() functions, may solve many problems. But i have no idea howto implement the garbage collector to detect pointers that got orphaned in a longjmp to automagically clean them. The next step would be to centralize global variables to clean them up more easily (as Glynn mentioned). and will not prevent QGIS from terminating or segfaulting eventually in case of a fatal error. Why? Because of the undefined state of global variables and structures and because of potential memory leaks that may sum up making QGIS unusable. Best regards Soeren Radim The QGIS GRASS data provider/plugin will be a clean C++ implementation using the thrift library. It will be better maintainable since developers do not need to know all the different
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
On Fri, Apr 18, 2014 at 2:19 PM, Sören Gebbert soerengebb...@googlemail.com wrote: Hi Radim, IMHO we GRASS developers are too stubborn to change the design of GRASS GIS to work as library with persistent applications. This would require rewriting plenty of core functionality and modification of all C-modules. But there is a solution for persistent applications, called Remote Procedure Call (RPC), more below. It sounds interesting but it seems to be a huge work an far future. I am also concerned about complexity, e.g. starting / keeping running servers on various platforms. I can imagine endless bug reports about server not running. I am looking for something which will not require permanent assistance. It is not KISS enough for me. Radim 2014-04-18 11:31 GMT+02:00 Radim Blazek radim.bla...@gmail.com: I have upgraded the vector provider to GRASS 7, layers may be added by drag from browser. The raster and the plugin are disabled. Be careful about multiple versions on the same system (LD_LIBRARY_PATH..., check with ldd if does not work). Unfortunately GRASS 7 moved ahead towards its aim to make life harder for anyone trying to use the GRASS libraries [1]. Basically more and more low level functions are calling exit() instead of returning error code if something failed. As I am not willing to implement GRASS module call for each simple function, we have to think again about hacks we can use: 1) add a requirement that GRASS 7 used with QGIS must be compiled with -fexceptions 2) add a requirement that a patch (it is a single line comment in fact) must be applied to GRASS 7 to make it usable with QGIS 3) use setjmp()/longjmp() 4) let QGIS crash whenever GRASS lib function fails I fear that none of these suggestions are good working solutions, since in case a fatal error or a segfault occurs nothing (no exception, no setjmp) will prevent QGIS from crashing. So i would like to suggest the following: 5.) Using a RPC interface for map metadata, vector and raster map access. Hence one or several GRASS server processes provide an RPC interface to access GRASS library functions that are needed in a persistent application. Most important is the vector editing, vector reading/writing, database access, raster reading and map metadata support. Everything else can be done using modules. Hence the RPC interface will support only a limited subset of the GRASS library functions. This RPC interface should only be used in the GRASS provider classes, the GRASS plugin itself can be written in C++ or Python. This approach will decouple GRASS from QGIS, since all communication is done via Inter Process Communication (IPC) using pipes or sockets. There is no need anymore to catch a fatal error or to link GRASS libraries directly to QGIS. The GRASS plugin can be implemented GRASS version independently and so the GRASS data provider, since they will not use GRASS functions directly only the RPC interface. The RPC implementation on the side of GRASS will provide a consistent interface that will not change in case the underlying GRASS API changes. I strongly suggest to use an existing RPC framework to implement fast binary data exchange and the IPC. My favorite is apache thrift[1], since it supports plenty of programming languages (C/C++, Python, Java, JavaScript, ...) and provides operating system independent client and server functionality. It supports exceptions on client side that can be emitted in case a GRASS server process died because of a fatal error, SIGINT or segfault. I am absolutely willing to implement the RPC server side in GRASS using thrift and C++, providing plenty of Python unit tests that show howto use it on the client side. We just need to decide what kind of GRASS library functionality is needed as RPC interface and what can be done in QGIS with C++/Python directly (gisrc creation, GRASS environmental variable settings, ...) or using GRASS modules (g.region, g.gisenv, ...). [1] https://thrift.apache.org/ Best regards Soeren Radim [1]https://trac.osgeo.org/grass/ticket/869#comment:1 On Thu, Mar 27, 2014 at 11:18 AM, Paolo Cavallini cavall...@faunalia.it wrote: Hi all. I learned during dinner that GRASS7 RC1 is due very soon. This opens the issue of its functioning in QGIS. IMHO: * the qgis-grass-plugin might stop working (this has to be tested) * some of the module options will be different * new modules will not be available in QGIS. I think we can deal with this in several ways: * dropping the plugin, and concentrate the work on Processing * upgrading both the plugin and Processing. In the first case, we have two major issues: * upgrading Processing GRASS modules * changing the current Processing behaviour, avoiding the import-export phase when piping consecutive GRASS commands; this makes GRASS modules slower than the equivalent commands of other backends. While the first issue can be solved easily by a
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
Hi Radim, 2014-04-22 16:03 GMT+02:00 Radim Blazek radim.bla...@gmail.com: On Fri, Apr 18, 2014 at 2:19 PM, Sören Gebbert soerengebb...@googlemail.com wrote: Hi Radim, IMHO we GRASS developers are too stubborn to change the design of GRASS GIS to work as library with persistent applications. This would require rewriting plenty of core functionality and modification of all C-modules. But there is a solution for persistent applications, called Remote Procedure Call (RPC), more below. It sounds interesting but it seems to be a huge work an far future. I Its far less work then modifying GRASS libraries to run with persistent applications. We need such an approach anyway, since vector editing is a GUI task - GUI is a persistent application - we need an exit safe interface to the GRASS libraries to implement a reliable vector editing GUI in GRASS itself. am also concerned about complexity, e.g. starting / keeping running servers on various platforms. I can imagine endless bug reports about server not running. I am looking for something which will not Since only QGIS is spawning the server process, it is always able to check if the process is running and can restart it when needed. We already make use of this concept in GRASS and it works nicely[1]. Spawning of processes and keeping track of them is an easy task with Qt, and it is completely OS independent. The Qt Bitcoin client for example makes use of RPC and runs on Windows, Mac OS and various Linux systems. So i don't see a problem here. require permanent assistance. It is not KISS enough for me. What would be a better KISS solution? IMO the RPC approach will make the implementation of the QGIS-GRASS data provider and the plugin simpler. There will be no mixing of C++ and C-code anymore. No setjmp/longjmp/exception workaround that is really complex and will not prevent QGIS from terminating or segfaulting eventually in case of a fatal error. The QGIS GRASS data provider/plugin will be a clean C++ implementation using the thrift library. It will be better maintainable since developers do not need to know all the different GRASS libraries (gis, raster, vector, database) but only the RPC interface that make use of a subset of the GRASS library functions, bundling them in an object oriented API. QGIS do not need to link against GRASS libraries anymore, future GRASS versions will be supported out of the box. This is much more KISS then the current approach. Best regards Soeren [1] http://grass.osgeo.org/programming7/namespacepython_1_1temporal_1_1c__libraries__interface.html Radim 2014-04-18 11:31 GMT+02:00 Radim Blazek radim.bla...@gmail.com: I have upgraded the vector provider to GRASS 7, layers may be added by drag from browser. The raster and the plugin are disabled. Be careful about multiple versions on the same system (LD_LIBRARY_PATH..., check with ldd if does not work). Unfortunately GRASS 7 moved ahead towards its aim to make life harder for anyone trying to use the GRASS libraries [1]. Basically more and more low level functions are calling exit() instead of returning error code if something failed. As I am not willing to implement GRASS module call for each simple function, we have to think again about hacks we can use: 1) add a requirement that GRASS 7 used with QGIS must be compiled with -fexceptions 2) add a requirement that a patch (it is a single line comment in fact) must be applied to GRASS 7 to make it usable with QGIS 3) use setjmp()/longjmp() 4) let QGIS crash whenever GRASS lib function fails I fear that none of these suggestions are good working solutions, since in case a fatal error or a segfault occurs nothing (no exception, no setjmp) will prevent QGIS from crashing. So i would like to suggest the following: 5.) Using a RPC interface for map metadata, vector and raster map access. Hence one or several GRASS server processes provide an RPC interface to access GRASS library functions that are needed in a persistent application. Most important is the vector editing, vector reading/writing, database access, raster reading and map metadata support. Everything else can be done using modules. Hence the RPC interface will support only a limited subset of the GRASS library functions. This RPC interface should only be used in the GRASS provider classes, the GRASS plugin itself can be written in C++ or Python. This approach will decouple GRASS from QGIS, since all communication is done via Inter Process Communication (IPC) using pipes or sockets. There is no need anymore to catch a fatal error or to link GRASS libraries directly to QGIS. The GRASS plugin can be implemented GRASS version independently and so the GRASS data provider, since they will not use GRASS functions directly only the RPC interface. The RPC implementation on the side of GRASS will provide a consistent interface that will not change in case the underlying GRASS API changes. I strongly suggest to use
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
Hi, 2014-04-20 1:34 GMT+02:00 Sören Gebbert soerengebb...@googlemail.com: Hence the wrapper will not only do simple function wrapping, it will also implement a higher level interface: [...] what about to create a trac wiki page dedicated to RFC and collect this notes there... ? Martin -- Martin Landa * http://geo.fsv.cvut.cz/gwiki/Landa ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
2014-04-19 0:59 GMT+02:00 Glynn Clements gl...@gclements.plus.com:
[...]
You're misunderstanding a key point.
Misunderstanding is usually a question of the point of view or level
of doggedness. Especially when someone is speaking about a key
point ;-)
Returning from G_fatal_error() won't leave GRASS libraries in an
unpredictable state. It's escaping from G_fatal_error() via e.g.
Returning from G_fatal_error() will typically result in a
(more-or-less immediate) segfault when the caller tries to use data
structures which contain garbage (because the callee never initialised
them, but raised a fatal error instead).
Well, and what do you think that I meant by unpredictable state?
The simplest example is G__malloc():
buf = malloc(n);
if (!buf) {
...
G_fatal_error(_(G_malloc: unable to allocate %lu bytes of memory at
%s:%d),
(unsigned long) n, file, line);
}
return buf;
If malloc() returns NULL and the subsequent G_fatal_error() call
returns, G__malloc() will return NULL to its caller. Which will
segfault as soon as it tries to store something in the allocated
memory.
You chose very specific case of course. There is probably something
you don't want to understand. In the core libraries (gis, vector,
raster, imagery) there are 553(!) places where G_fatal_error() is
called. Some time ago you introduced G_fatal_error() even to the
function which tries to open raster map [1]. So if the raster map
doesn't exists the function is calling exit()! This is something which
makes life harder for anyone who want to use GRASS libraries (as you
already mentioned) with one exception - modules - standalone programs,
where you don't need to specify `if` statement. You can hardly explain
to someone that our libraries call fatal exit even when raster map is
not found. Yes, you can if don't want to change your opinion
regardless any input from outside world. For such cases, like opening
raster map would G_set_fatal_error() simply make sense.
It achieves this by calling exit() (or optionally raising SIGSEGV; why
it didn't use SIGABRT, I have no idea). Calling longjmp() would work.
If it was C++, raising an exception would work. Returning will *not*
work.
[...]
Martin
[1]
http://trac.osgeo.org/grass/changeset/40209/grass/trunk/lib/raster/opencell.c
--
Martin Landa * http://geo.fsv.cvut.cz/gwiki/Landa
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
Hi Martin, On Fri, 18. Apr 2014 at 12:20:15 +0200, Martin Landa wrote: [...] I still think that we should provide in GRASS API function like G_set_fatal_error() with default value G_FATAL_ERROR_EXIT (current behaviour). The second option would be G_FATAL_ERROR_RETURN with big big warning in the API manual that you are living GRASS libraries in completely unpredictable state and you should immediately stop an application which is using GRASS libraries. I also don't think that would help much. You would still need to use G_set_error_routine to track that there was an fatal error and in turn avoid a crash by not touching the result when the call returns - and keep away from calling other library function because of the unpredictable state of the library. On Fri, 18. Apr 2014 at 23:59:25 +0100, Glynn Clements wrote: It achieves this by calling exit() (or optionally raising SIGSEGV; why it didn't use SIGABRT, I have no idea). Calling longjmp() would work. If it was C++, raising an exception would work. Returning will *not* work. I guess there are also library calls inside the libraries themselves and those would not be aware that there is a need to handle invalid results and might crash before the toplevel caller has a chance know of the error. I think that's Glynn's keypoint. Using setjmp()/longjmp() or throwing C++ exceptions (with the libraries built with -fexceptions) in the error routine would take you back to the caller that is aware of the error. That avoids the crash, but also leaves the library in an unpredictable state. To avoid that there's no other way than returning and handling all the errors properly. Probably huge amount of work with not much gain for GRASS. The RPC approach will IMHO be less work and also cause less friction. Jürgen -- Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31 Dipl.-Inf. (FH) Rheinstraße 13Fax. +49-4931-918175-50 Software Engineer D-26506 Norden http://www.norbit.de QGIS PSC member (RM) Germany IRC: jef on FreeNode -- norBIT Gesellschaft fuer Unternehmensberatung und Informationssysteme mbH Rheinstrasse 13, 26506 Norden GF: Jelto Buurman, HR: Amtsgericht Emden, HRB 5502 ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
Martin Landa wrote: Returning from G_fatal_error() won't leave GRASS libraries in an unpredictable state. It's escaping from G_fatal_error() via e.g. Returning from G_fatal_error() will typically result in a (more-or-less immediate) segfault when the caller tries to use data structures which contain garbage (because the callee never initialised them, but raised a fatal error instead). Well, and what do you think that I meant by unpredictable state? I assumed that you meant the same thing that I do when I point out that longjmp()ing out of a fatal error handling will leave the libraries' data in an unpredictable state, i.e. that a *subsequent* call to a GRASS library function may result in a crash. Having G_fatal_error() return will usually cause an immediate crash, without needed any subsequent calls. The simplest example is G__malloc(): You chose very specific case of course. I chose the simplest example. But the more general case is: 1. Code performs a task which can either fail or succeed. 2a. On success, some form of result is stored in one or more variables. 2b. On failure, the code calls G_fatal_error(). 3. The code uses the values of the variables assigned in 2a, taking it for granted that they contain valid values (because if 1 had failed, the G_fatal_error() call in 2b ensures that we wouldn't be here). If G_fatal_error() can return, practically any function which calls it will either attempt to use garbage data, or will (silently) return garbage data to its caller. There is probably something you don't want to understand. In the core libraries (gis, vector, raster, imagery) there are 553(!) places where G_fatal_error() is called. Some time ago you introduced G_fatal_error() even to the function which tries to open raster map [1]. So if the raster map doesn't exists the function is calling exit()! This is something which makes life harder for anyone who want to use GRASS libraries (as you already mentioned) with one exception - modules - standalone programs, Modules aren't an exception, they're the rule. People wanting to use GRASS libraries in a persistent application are the exception. It's the latter category that's doing the not understanding. Specifically, they don't understand the magnitude of the cost involved in propagating errors all the way to the top-most level using status returns at each step. This cost is why modern languages have exceptions. Unfortunately: 1. C doesn't have exceptions. The closest thing it has is setjmp/longjmp. 2. Exceptions eliminate much of the cost, but not all of it. You still have to write exception-safe code. But the difference is significant enough that I wouldn't object to the (comparatively minor) added complexity resulting from making code exception-safe. Apart from anything else, the added complexity would be limited to the implementation of library functions; it doesn't impose a unreasonable burden on the callers in the way that using status returns does. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
Vaclav Petras wrote: The key feature of G_fatal_error() isn't printing error messages. The key feature is that it doesn't return. This is one of the problems of G_fatal_error(), it does not gives good error messages to the user. User gets G_malloc: unable to allocate %lu bytes of memory at %s:%d but does not have any idea when and why this happened and what to do about it. If the module calling G_malloc() had the chance to report error by itself, it could suggest to user to change some parameter or use a smaller map. It could, but 99% of the time it won't. Instead, it will either assume that the returned pointer is valid, or provide some vacuous message such as out of memory (without the reference to the source file, line number or requested size that is given at present). So in a handful of cases, you'd get a better error message, while the rest will be even worse. At least the current situation ensures that there are enough clues that the GRASS developers can provide assistance to the user upon request. When I replaced status returns with fatal errors in a number of core functions (r40209, r40217, r40701), in some of the cases where the function's return type changed from int to void, not a single error was generated; IOW, not one of the callers actually bothered to check whether the call succeeded. Similarly, in cases where an int/pointer return value could be negative or null to indicate an error, the cases where the (now redundant) error tests needed to be removed from the callers were often a minority, sometimes a very small minority, occasionally zero. With G_fatal_error(), the requirement for the caller to check for errors is removed, and with it the possibility that the caller will either fail to check for errors, or produce an entirely useless error message (although I think that we've now eliminated all of the G_fatal_error(oops!) cases; no, I'm not joking). G_fatal_error() is too low level not only for messages but in general, for Python libraries wrapping GRASS library, it does not allow to throw an exception in case of an error. And the same applies for C++ wrappers. If you want to turn fatal errors into exceptions, the correct way to do it is to use setjmp/longjmp (and contribute patches to make any intervening code exception-safe). -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
G_fatal_error() is too low level not only for messages but in general, for Python libraries wrapping GRASS library, it does not allow to throw an exception in case of an error. And the same applies for C++ wrappers. If you want to turn fatal errors into exceptions, the correct way to do it is to use setjmp/longjmp (and contribute patches to make any intervening code exception-safe). As I wrote before, I hope that RPC approach suggested here by Soeren will work for languages with exceptions and for persistent applications while GRASS modules will benefit from the advantages of GRASS library with G_fatal_error() with exit(). Vaclav ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
Hi, 2014-04-18 11:31 GMT+02:00 Radim Blazek radim.bla...@gmail.com: [] Unfortunately GRASS 7 moved ahead towards its aim to make life harder for anyone trying to use the GRASS libraries [1]. Basically more and personally I am not fan of this approach. I still think that we should provide in GRASS API function like G_set_fatal_error() with default value G_FATAL_ERROR_EXIT (current behaviour). The second option would be G_FATAL_ERROR_RETURN with big big warning in the API manual that you are living GRASS libraries in completely unpredictable state and you should immediately stop an application which is using GRASS libraries. There no so big chance that someone will rewrite GRASS libraries to be suitable for long running applications, on the other hand we should allow in API possibility to avoid calling exit with big warning to the programmer. Than it will be his/her responsibility if he/she uses G_FATAL_ERROR_RETURN. Martin ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
Hi Radim, IMHO we GRASS developers are too stubborn to change the design of GRASS GIS to work as library with persistent applications. This would require rewriting plenty of core functionality and modification of all C-modules. But there is a solution for persistent applications, called Remote Procedure Call (RPC), more below. 2014-04-18 11:31 GMT+02:00 Radim Blazek radim.bla...@gmail.com: I have upgraded the vector provider to GRASS 7, layers may be added by drag from browser. The raster and the plugin are disabled. Be careful about multiple versions on the same system (LD_LIBRARY_PATH..., check with ldd if does not work). Unfortunately GRASS 7 moved ahead towards its aim to make life harder for anyone trying to use the GRASS libraries [1]. Basically more and more low level functions are calling exit() instead of returning error code if something failed. As I am not willing to implement GRASS module call for each simple function, we have to think again about hacks we can use: 1) add a requirement that GRASS 7 used with QGIS must be compiled with -fexceptions 2) add a requirement that a patch (it is a single line comment in fact) must be applied to GRASS 7 to make it usable with QGIS 3) use setjmp()/longjmp() 4) let QGIS crash whenever GRASS lib function fails I fear that none of these suggestions are good working solutions, since in case a fatal error or a segfault occurs nothing (no exception, no setjmp) will prevent QGIS from crashing. So i would like to suggest the following: 5.) Using a RPC interface for map metadata, vector and raster map access. Hence one or several GRASS server processes provide an RPC interface to access GRASS library functions that are needed in a persistent application. Most important is the vector editing, vector reading/writing, database access, raster reading and map metadata support. Everything else can be done using modules. Hence the RPC interface will support only a limited subset of the GRASS library functions. This RPC interface should only be used in the GRASS provider classes, the GRASS plugin itself can be written in C++ or Python. This approach will decouple GRASS from QGIS, since all communication is done via Inter Process Communication (IPC) using pipes or sockets. There is no need anymore to catch a fatal error or to link GRASS libraries directly to QGIS. The GRASS plugin can be implemented GRASS version independently and so the GRASS data provider, since they will not use GRASS functions directly only the RPC interface. The RPC implementation on the side of GRASS will provide a consistent interface that will not change in case the underlying GRASS API changes. I strongly suggest to use an existing RPC framework to implement fast binary data exchange and the IPC. My favorite is apache thrift[1], since it supports plenty of programming languages (C/C++, Python, Java, JavaScript, ...) and provides operating system independent client and server functionality. It supports exceptions on client side that can be emitted in case a GRASS server process died because of a fatal error, SIGINT or segfault. I am absolutely willing to implement the RPC server side in GRASS using thrift and C++, providing plenty of Python unit tests that show howto use it on the client side. We just need to decide what kind of GRASS library functionality is needed as RPC interface and what can be done in QGIS with C++/Python directly (gisrc creation, GRASS environmental variable settings, ...) or using GRASS modules (g.region, g.gisenv, ...). [1] https://thrift.apache.org/ Best regards Soeren Radim [1]https://trac.osgeo.org/grass/ticket/869#comment:1 On Thu, Mar 27, 2014 at 11:18 AM, Paolo Cavallini cavall...@faunalia.it wrote: Hi all. I learned during dinner that GRASS7 RC1 is due very soon. This opens the issue of its functioning in QGIS. IMHO: * the qgis-grass-plugin might stop working (this has to be tested) * some of the module options will be different * new modules will not be available in QGIS. I think we can deal with this in several ways: * dropping the plugin, and concentrate the work on Processing * upgrading both the plugin and Processing. In the first case, we have two major issues: * upgrading Processing GRASS modules * changing the current Processing behaviour, avoiding the import-export phase when piping consecutive GRASS commands; this makes GRASS modules slower than the equivalent commands of other backends. While the first issue can be solved easily by a couple of people in a few days, the second one is more tricky, and requires hard coding skills. In addition, we'll no longer be able to edit GRASS vectors directly. In the second case, we'll have more work, and a not-so-nice duplication. I would like to have an open discussion on this, avoiding things to just happen, with the possible negative consequences. All the best. -- Paolo Cavallini - www.faunalia.eu QGIS PostGIS courses:
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
Martin Landa wrote:
Unfortunately GRASS 7 moved ahead towards its aim to make life harder
for anyone trying to use the GRASS libraries [1]. Basically more and
personally I am not fan of this approach. I still think that we should
provide in GRASS API function like G_set_fatal_error() with default
value G_FATAL_ERROR_EXIT (current behaviour). The second option would
be G_FATAL_ERROR_RETURN with big big warning in the API manual that
you are living GRASS libraries in completely unpredictable state
You're misunderstanding a key point.
Returning from G_fatal_error() won't leave GRASS libraries in an
unpredictable state. It's escaping from G_fatal_error() via e.g.
longjmp() (so that control doesn't return to the caller) that will do
that.
Returning from G_fatal_error() will typically result in a
(more-or-less immediate) segfault when the caller tries to use data
structures which contain garbage (because the callee never initialised
them, but raised a fatal error instead).
The simplest example is G__malloc():
buf = malloc(n);
if (!buf) {
...
G_fatal_error(_(G_malloc: unable to allocate %lu bytes of memory at
%s:%d),
(unsigned long) n, file, line);
}
return buf;
If malloc() returns NULL and the subsequent G_fatal_error() call
returns, G__malloc() will return NULL to its caller. Which will
segfault as soon as it tries to store something in the allocated
memory.
The key feature of G_fatal_error() isn't printing error messages. The
key feature is that it doesn't return.
It achieves this by calling exit() (or optionally raising SIGSEGV; why
it didn't use SIGABRT, I have no idea). Calling longjmp() would work.
If it was C++, raising an exception would work. Returning will *not*
work.
--
Glynn Clements gl...@gclements.plus.com
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
On Fri, Apr 18, 2014 at 6:59 PM, Glynn Clements gl...@gclements.plus.comwrote: The key feature of G_fatal_error() isn't printing error messages. The key feature is that it doesn't return. This is one of the problems of G_fatal_error(), it does not gives good error messages to the user. User gets G_malloc: unable to allocate %lu bytes of memory at %s:%d but does not have any idea when and why this happened and what to do about it. If the module calling G_malloc() had the chance to report error by itself, it could suggest to user to change some parameter or use a smaller map. With G_fatal_error() user has only the possibility to try to guess from debug messages, ask on mailing list or keep trying with random changes in input parameters. I don't believe that this is efficient. G_fatal_error() is too low level not only for messages but in general, for Python libraries wrapping GRASS library, it does not allow to throw an exception in case of an error. And the same applies for C++ wrappers. As far as I understand, the RPC wrapper suggested earlier by Soeren, would hopefully allow to have exceptions (and no exit()) in Python and C++ and thus potentially higher level error messages and also general error handling would be possible (for those not using the original library). ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
On 28/03/14 15:16, Paolo Cavallini wrote: Il 28/03/2014 15:04, Martin Dobias ha scritto: On Fri, Mar 28, 2014 at 11:48 AM, Paolo Cavallini cavall...@faunalia.it wrote: * to add GRASS browsing capabilities in QGIS file browser The support for GRASS is already in the browser - if you enter a directory that is a GRASS database, it will detect it and show locations/mapsets/maps/layers. That's great news, never tried. Works beautifully, thanks. I cannot reproduce this. How does it work exactly ? When I browse into a GRASS database directory, it just shows up like any other directory. Moritz ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
On Tue, Apr 8, 2014 at 2:18 PM, Moritz Lennert mlenn...@club.worldonline.be wrote: On 28/03/14 15:16, Paolo Cavallini wrote: Il 28/03/2014 15:04, Martin Dobias ha scritto: On Fri, Mar 28, 2014 at 11:48 AM, Paolo Cavallini cavall...@faunalia.it wrote: * to add GRASS browsing capabilities in QGIS file browser The support for GRASS is already in the browser - if you enter a directory that is a GRASS database, it will detect it and show locations/mapsets/maps/layers. That's great news, never tried. Works beautifully, thanks. I cannot reproduce this. How does it work exactly ? When I browse into a GRASS database directory, it just shows up like any other directory. The GRASS location is added as another item with GRASS icon in the tree on the same level where is the location dir. It means that the location will appear twice, first as regular dir and then as a GRASS location. The items do not seem to be sorted by name, so maybe you have to scroll down. Radim Moritz ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
Radim Blazek radim.bla...@gmail.com writes: On Tue, Apr 8, 2014 at 2:18 PM, Moritz Lennert mlenn...@club.worldonline.be wrote: On 28/03/14 15:16, Paolo Cavallini wrote: Il 28/03/2014 15:04, Martin Dobias ha scritto: On Fri, Mar 28, 2014 at 11:48 AM, Paolo Cavallini cavall...@faunalia.it wrote: * to add GRASS browsing capabilities in QGIS file browser The support for GRASS is already in the browser - if you enter a directory that is a GRASS database, it will detect it and show locations/mapsets/maps/layers. That's great news, never tried. Works beautifully, thanks. I cannot reproduce this. How does it work exactly ? When I browse into a GRASS database directory, it just shows up like any other directory. The GRASS location is added as another item with GRASS icon in the tree on the same level where is the location dir. It means that the location will appear twice, first as regular dir and then as a GRASS location. The items do not seem to be sorted by name, so maybe you have to scroll down. Hm - I think I also need a step-by-step guide. I am using QGIS 2.2.0 on OSX Maverick and I don't get it to work. 0) Open QGIS 1) I open the browser (View - Panels - Browser) 2) Then I go via the Home folder (top entry) to a GRASS database 3) and now, where should I see what? What is the root in the tree structure? I can only see the folder structure. Rainer Radim Moritz ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev -- Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany) Centre of Excellence for Invasion Biology Stellenbosch University South Africa Tel : +33 - (0)9 53 10 27 44 Cell: +33 - (0)6 85 62 59 98 Fax : +33 - (0)9 58 10 27 44 Fax (D):+49 - (0)3 21 21 25 22 44 email: rai...@krugs.de Skype: RMkrug PGP: 0x0F52F982 ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
On Tue, Apr 8, 2014 at 3:54 PM, Rainer M Krug rai...@krugs.de wrote: Radim Blazek radim.bla...@gmail.com writes: On Tue, Apr 8, 2014 at 2:18 PM, Moritz Lennert mlenn...@club.worldonline.be wrote: On 28/03/14 15:16, Paolo Cavallini wrote: Il 28/03/2014 15:04, Martin Dobias ha scritto: On Fri, Mar 28, 2014 at 11:48 AM, Paolo Cavallini cavall...@faunalia.it wrote: * to add GRASS browsing capabilities in QGIS file browser The support for GRASS is already in the browser - if you enter a directory that is a GRASS database, it will detect it and show locations/mapsets/maps/layers. That's great news, never tried. Works beautifully, thanks. I cannot reproduce this. How does it work exactly ? When I browse into a GRASS database directory, it just shows up like any other directory. The GRASS location is added as another item with GRASS icon in the tree on the same level where is the location dir. It means that the location will appear twice, first as regular dir and then as a GRASS location. The items do not seem to be sorted by name, so maybe you have to scroll down. Hm - I think I also need a step-by-step guide. I am using QGIS 2.2.0 on OSX Maverick and I don't get it to work. 0) Open QGIS 1) I open the browser (View - Panels - Browser) 2) Then I go via the Home folder (top entry) to a GRASS database 3) and now, where should I see what? What is the root in the tree structure? I can only see the folder structure. Probably a bug. It is checking for location/PERMANENT/DEFAULT_WIND. You have maybe lower case names? If so, can you test with upper? Please fill a new issue. Radim Rainer Radim Moritz ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev -- Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany) Centre of Excellence for Invasion Biology Stellenbosch University South Africa Tel : +33 - (0)9 53 10 27 44 Cell: +33 - (0)6 85 62 59 98 Fax : +33 - (0)9 58 10 27 44 Fax (D):+49 - (0)3 21 21 25 22 44 email: rai...@krugs.de Skype: RMkrug PGP: 0x0F52F982 ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
On 08/04/14 15:31, Radim Blazek wrote: On Tue, Apr 8, 2014 at 2:18 PM, Moritz Lennert mlenn...@club.worldonline.be wrote: On 28/03/14 15:16, Paolo Cavallini wrote: Il 28/03/2014 15:04, Martin Dobias ha scritto: On Fri, Mar 28, 2014 at 11:48 AM, Paolo Cavallini cavall...@faunalia.it wrote: * to add GRASS browsing capabilities in QGIS file browser The support for GRASS is already in the browser - if you enter a directory that is a GRASS database, it will detect it and show locations/mapsets/maps/layers. That's great news, never tried. Works beautifully, thanks. I cannot reproduce this. How does it work exactly ? When I browse into a GRASS database directory, it just shows up like any other directory. The GRASS location is added as another item with GRASS icon in the tree on the same level where is the location dir. It means that the location will appear twice, first as regular dir and then as a GRASS location. The items do not seem to be sorted by name, so maybe you have to scroll down. That was it. I just never thought to go down further... Thanks ! Moritz ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
One last addition to this from my side:
Since, as I mentioned, we have evaluated many of the
issues discussed in this thread in the design of the
GRASS plug-in for SEXTANTE/gvSIG CE, here are some things
you might want to take into consideration:
On 28/03/14 11:48, Paolo Cavallini wrote:
Il 28/03/2014 08:34, Paolo Cavallini ha scritto:
Thanks to all for the interesting comments. Obviously we have a varied
and rich ecosystem here, that's why I want to preserve it.
Obviously we all want to have all. IMHO the points are:
* can we (GRASS+QGIS) agree on a specific course of actions?
* can we find the resource to implement it?
Of course we could think of sticking with GRASS6, but I think sooner or
later this will be untenable (e.g. major distros will switch to GRASS7
once out), so better prepare now.
Quick interesting chat with Luca Delucchi and Martin Landa from GRASS
dev team.
One interesting option would be:
* to add GRASS browsing capabilities in QGIS file browser
Browsing capabilities for existing GRASS mapsets would be
nice, but having the plug-in directly manipulate the data
in there leads down a difficult path (see my notes further
below).
* to add support to direct GRASS reading (possibly writing) in
Processing, and avoid import/export whenever possible; v.external and
v.out.external could be used for this; this is probably the thoughest part
As far as I understand, the v.out.* modules create read-only
datasets. They also create minimal topological data structures
that are not enough for many GRASS modules (correct me if any of
these statements is no longer true). Therefore, we decided against
this path and use v.in/out.ogr instead -- with all known problems.
But r.external seems to work fine, so that at least raster I/O is
efficient enough; with the one limitation being that GRASS does not
support multi-band rasters in single files (we do not have a good
solution for working around that problem yet).
* of course modules should be upgraded anyway
* GRASS direct digitizing can be skipped; heavy digitizer can use GRASS
directly
(see below)
* keep a GRASS shell?
This only works well with the original GRASS plug-in design, not
with the one derived from SEXTANTE:
Any plug-in design that allows direct manipulation of existing
GRASS data and/or runs GRASS modules on existing mapsets is
not very compatible with the SEXTANTE/Processing on-the-fly
import/process/export/delete temp approach. As long as the GRASS
mapset is guaranteed to be temporary, the above chain is clean
and simple. But if you give the plug-in access to existing mapset
data, then you need a nice, thick wrapper of additional safety
checks to make sure that the final delete step *never* deletes
any pre-existing user data! Not only would you have to keep track
of each and every bit of data that an intermediate processing step
creates (modules that call other modules!), but also make sure
that no stale data is left in the user's mapset if processing does
not complete.
This is all very, very tricky and normally the role of the GRASS user
who knows about GRASS data management and is in full control.
I.e. if you want to make this safe, then you have to expose the GRASS
mapset and management completely to the user -- and this in turn
defeats the idea of radically simplifying the use of GRASS modules.
This is the reason why in gvSIG CE we steered clear of existing
user mapsets; our conviction being that data safety is always more
important than convenience and that advanced users can/will just run
GRASS by themselves.
Best,
Ben
* with all the above, upgrading the plugin may probably be skipped.
Any thoughts.
--
Dr. Benjamin Ducke, M.A.
{*} Geospatial Consultant
{*} GIS Developer
bendu...@fastmail.fm
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
Il 28/03/2014 15:16, Paolo Cavallini ha scritto: Il 28/03/2014 15:04, Martin Dobias ha scritto: The support for GRASS is already in the browser - if you enter a directory that is a GRASS database, it will detect it and show locations/mapsets/maps/layers. In the GRASS plugin there are a few more features, i.e.: * showing history (important) * copy/rename/delete. If we can add these, I think the corresponding section of the plugin would be rather useless. Am I missing something? What about writing down the minumum set of requirements to be preserved? Please add it to this thread, we can note it on a wiki page once bolied down. All the best. -- Paolo Cavallini - www.faunalia.eu QGIS PostGIS courses: http://www.faunalia.eu/training.html ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
Il 27/03/2014 12:33, Nathan Woodrow ha scritto: I would vote for dropping the plugin and just updating the processing plugin. Having both ways is bad for us and bad for users, even worse when some functions are missing from one but not in the other. I understand well the point; however, the plugin has additional functions, e.g.: * a grass shell * a grass data browser * a grass digitizing environment. Whether these are important or not, it's a matter of users. All the best. -- Paolo Cavallini - www.faunalia.eu QGIS PostGIS courses: http://www.faunalia.eu/training.html ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
Dear all, From my user perspective (I am using both GRASS and QGIS, or the other way around, depends on topic), processing is not really a replacement for the GRASS plugin. It is handy and probably better for those who do not use GRASS but are interested in single functions / modules. Yet, once you have a GRASS database with a considerable amount of data, the GRASS plugin is very valuable for accessing those data in QGIS (e.g. for cartography). Also users with no or little GRASS experience benefit from the GRASS plugin in cases where they have access to a GRASS database. That way they can work on it without acquainting themselves with a new GIS in depth... Concluding, if you have the possibility to maintain it, keep the GRASS plugin in QGIS! Cheers Stefan -Original Message- From: qgis-developer-boun...@lists.osgeo.org [mailto:qgis-developer-boun...@lists.osgeo.org] On Behalf Of Paolo Cavallini Sent: 27. mars 2014 12:41 To: Nathan Woodrow Cc: qgis-developer; grass-dev Subject: Re: [Qgis-developer] GRASS QGIS: the future Il 27/03/2014 12:33, Nathan Woodrow ha scritto: I would vote for dropping the plugin and just updating the processing plugin. Having both ways is bad for us and bad for users, even worse when some functions are missing from one but not in the other. I understand well the point; however, the plugin has additional functions, e.g.: * a grass shell * a grass data browser * a grass digitizing environment. Whether these are important or not, it's a matter of users. All the best. -- Paolo Cavallini - www.faunalia.eu QGIS PostGIS courses: http://www.faunalia.eu/training.html ___ Qgis-developer mailing list qgis-develo...@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
That sounds very reasonable to me. Proper GDAL/OGR handling for GRASS 7 would be very nice in any case (see http://trac.osgeo.org/gdal/ticket/2953). As for a GRASS data browser, I think, a plugin would be required with regards to user friendliness, because one needs to know what files to access from a GRASS data base when using GDAL. I also understand that at some point in time one will have to use GRASS directly in order to access full functionality (e.g. ortho-rectification, nviz, mapswipe, animation and stuff), which makes the way Moritz suggests maybe even more reasonable... Cheers Stefan -Original Message- From: Moritz Lennert [mailto:mlenn...@club.worldonline.be] Sent: 27. mars 2014 13:36 To: Blumentrath, Stefan Cc: Paolo Cavallini; Nathan Woodrow; qgis-developer; grass-dev Subject: Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future I'm not much of a QGIS-as-GRASS-frontend user, either, but from a general point of view I also think that duplication is a problem and that the current way to go in QGIS is the processing framework. So; On 27/03/14 12:49, Blumentrath, Stefan wrote: I understand well the point; however, the plugin has additional functions, e.g.: * a grass shell couldn't this be implemented within the processing environment ? * a grass data browser If we are talking about accessing GRASS data for loading into QGIS, wouldn't it be a better idea to improve the GDAL/OGR handling in order to be able to load GRASS data just like any other data format ? * a grass digitizing environment. Maybe this could be split out into a specific plugin ? Moritz ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
On Thu, Mar 27, 2014 at 9:38 AM, Blumentrath, Stefan stefan.blumentr...@nina.no wrote: I also understand that at some point in time one will have to use GRASS directly in order to access full functionality (e.g. ortho-rectification, nviz, mapswipe, animation and stuff), which makes the way Moritz suggests maybe even more reasonable... I hope that we find the way to start these tools from QGIS. Tck/Tk nviz is starting in this way. I hope that it will be possible to do the same for g.gui.* modules. Also, it would be nice if the QGIS GRASS vector digitizer (plugin/tool) would use the same backend as vector digitizer in GRASS wxGUI (Python subprocessing exit-safe wrapper). Now the first problem is that there is no reusable backend in wxGUI. Vaclav ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
In the gvSIG CE project, we had to make the same
type of decisions and came to our own conclusions.
Allow me to summarize our reasoning, maybe it will be
useful for the QGIS project, as well:
1. The number one cause of irritations among novice
users is having to set up a GRASS mapset and having
to understand how GRASS data management works.
2. The number two cause of irritations are the effects
of importing vector data with bad topology into a
GRASS mapset.
3. The original QGIS plugin does nothing to alleviate (1).
No plug-in, however cleverly designed, can do anything
about (2): garbage in, garbage out.
4. GRASS power users gain very little (if anything)
from running GRASS through a host GIS, such as QGIS
or gvSIG. But they do lose functionality, such as the
d.* family of modules.
Therefore, we gave up trying to design a plug-in for
advanced users. We assume that such users will use
GRASS through its native CLI and/or native GUI.
The resulting design of the original SEXTANTE-GRASS
interface (which is now also mirrored in the Python
re-write that became QGIS' Processing) addresses users
that either don't care much for GRASS' CLI capabilities
and internal data management, or are willing to switch to
native GRASS as needed.
If you want to change this and address another type
of user, then you will need to re-examine the entire design
of the current SEXTANTE/Processing approach, which is to use
only temporary mapsets and perform data import/export every
time a GRASS module is run.
You can optimize the I/O performance of Processing by using
r.external to directly access raster input maps. But there
is little you can do about vector data with the current
design, as GRASS needs to build its own topological data
structures (and rebuild them every time you run a GRASS module
on non-topological input!).
In any case, I do not think that it is worth maintaining
the original QGIS plugin, since it is neither very well
suited for novice nor advanced users, IMHO.
Best,
Ben
On 27/03/14 14:38, Blumentrath, Stefan wrote:
That sounds very reasonable to me.
Proper GDAL/OGR handling for GRASS 7 would be very nice in any case (see
http://trac.osgeo.org/gdal/ticket/2953).
As for a GRASS data browser, I think, a plugin would be required with
regards to user friendliness, because one needs to know what files to access
from a GRASS data base when using GDAL.
I also understand that at some point in time one will have to use GRASS
directly in order to access full functionality (e.g. ortho-rectification,
nviz, mapswipe, animation and stuff), which makes the way Moritz suggests
maybe even more reasonable...
Cheers
Stefan
-Original Message-
From: Moritz Lennert [mailto:mlenn...@club.worldonline.be]
Sent: 27. mars 2014 13:36
To: Blumentrath, Stefan
Cc: Paolo Cavallini; Nathan Woodrow; qgis-developer; grass-dev
Subject: Re: [GRASS-dev] [Qgis-developer] GRASS QGIS: the future
I'm not much of a QGIS-as-GRASS-frontend user, either, but from a general
point of view I also think that duplication is a problem and that the current
way to go in QGIS is the processing framework. So;
On 27/03/14 12:49, Blumentrath, Stefan wrote:
I understand well the point; however, the plugin has additional functions,
e.g.:
* a grass shell
couldn't this be implemented within the processing environment ?
* a grass data browser
If we are talking about accessing GRASS data for loading into QGIS, wouldn't
it be a better idea to improve the GDAL/OGR handling in order to be able to
load GRASS data just like any other data format ?
* a grass digitizing environment.
Maybe this could be split out into a specific plugin ?
Moritz
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
--
Dr. Benjamin Ducke, M.A.
{*} Geospatial Consultant
{*} GIS Developer
bendu...@fastmail.fm
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
