Re: [GRASS-dev] [GRASS GIS] #2134: Create a general exit-safe interface to C libraries
#2134: Create a general exit-safe interface to C libraries
--+-
Reporter: wenzeslaus| Owner: grass-dev@…
Type: enhancement | Status: new
Priority: normal| Milestone: 7.0.0
Component: Python ctypes | Version: svn-trunk
Keywords: G_fatal_error, exit, multiprocessing |Platform: All
Cpu: Unspecified |
--+-
Comment(by huhabla):
Replying to [comment:9 wenzeslaus]:
Replying to [comment:6 glynn]:
Replying to [comment:4 zarch]:
Or as already suggested by Glynn (#1646) wrap the
G_add_error_handler,
Using an error handler allows you to avoid process termination. But
once a fatal error has occurred, you cannot safely call any GRASS
function; doing so may well result in a segfault.
One of the issues which `G_add_error_handler` is trying to solve is to
provide meaningful error message to the user. For example, failing to open
some temporary file causes `exit` with `No such file /tmp/kjewbf8d38dj`.
This does not help user nor programmer to understand that the error
occurred (when does this happened, what is the stack trace, what are
consequences and what are suggestions to solve it). In other words,
sometimes the message provided by `G_fatal_error` caller is too low level.
Python `RPCServer` with wrapper functions throwing exceptions would help
to solve this issue. But it seems to me that #1646 remains valid for
pyGRASS (and possibly others) and C code itself.
I have re-designed the RPC interface, now the Python function wrapper will
return an exception and the result of the function calls, so that the RPC
server interface that provides the {{{call()}}} functions can raise these
exceptions (exceptions raised in the subprocess will kill the subprocess
and will not be catch'd in the parent process). Hence, the Python wrapper
functions transform the C-function return values into meaningful
exceptions that will be raised in the parent process.
While re-designing i concluded that a no wait function call
{{{call_no_wait()}}} is not meaningful when mixed with calls that wait to
receive data. There is only a limited number of C-functions that do not
return values or return states. It is better to wait for a function call
to finish, than risking a race condition in case a fatal error occur'd
meanwhile. An exception is the messaging interface, which should stay as
is.
However, maybe two RPC interfaces are meaningful: one that waits for
functions to return (expecting return values including exceptions) and one
that does not wait?
Replying to [comment:7 huhabla]:
In my opinion the RPC approach is only meaningful for persistent
applications that need fast access to C-library functions, or that need
low level API access for data modification (like digitizing).
And this is not only vector and raster digitizing, this is also new
scatter plot tool and in fact the whole `g.gui.iclass`, `nviz` (which is
unfortunately more complicated) and of course everything temporal-related
(everything started to be temporal-related).
My intention to write the RPC server was to make the temporal
framework usable in persistent applications and to be as fast as possible.
I'm not sure how the speed of `RPCServer` compares to module call but
the speed is not the only advantage. Fine control of what is called and a
smoother interface (possibly, depending on wrappers) is the other
advantage. Calling subprocess from GUI for every single task and parsing
its output is cumbersome.
I have added benchmark runs to the rpc server script, to get an idea what
the performance loss and gain of the RPC interface is:
{{{
GRASS 7.0.svn (Test XY):~ python c_library_interface.py
##
TESTS
ERROR: A fatal error
WARNING:root:Needed to restart the rpc server
ERROR: A fatal error
WARNING:root:Needed to restart the rpc server
##
##
Raster map exists benchmark
Time to call 1000 functions directly: 0.017043s
Time to call 1000 functions via RPC: 0.178600s
Time to perform 1000 g.findfile module runs: 30.343877s
##
##
Raster map info benchmark
Time to call 1 functions directly: 0.856104s
Time to call 1 functions via RPC: 7.189188s
Time to perform 1 r.info module runs: 120.261683s
##
}}}
Re: [GRASS-dev] new flag to interpolate from nearby raster cells for v.what.rast
On Thu, Nov 14, 2013 at 11:15 AM, Hamish hamis...@yahoo.com wrote: Hi, I just added in the dev branches two new flags for v.what.rast. [...] The second flag changes the default containing-grid-cell method to a weighted avg. of the 4 nearest raster cell centers (IDW). The search radius is not very big [...] AFAIK, IDW takes as argument the radius which is fixed to 1 which is questionable. Since v.what.rast samples a raster value at a given location, why not use the standard raster sampling methods nearest: nearest neighbor linear: linear interpolation cubic: cubic convolution lanczos: lanczos filter linear_f: linear interpolation with fallback cubic_f: cubic convolution with fallback lanczos_f: lanczos filter with fallback which are used by r.proj and i.rectify? The concept is essentially the same: for a given location, estimate the corresponding raster value. Markus M ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [GRASS GIS] #2134: Create a general exit-safe interface to C libraries
#2134: Create a general exit-safe interface to C libraries
--+-
Reporter: wenzeslaus| Owner: grass-dev@…
Type: enhancement | Status: new
Priority: normal| Milestone: 7.0.0
Component: Python ctypes | Version: svn-trunk
Keywords: G_fatal_error, exit, multiprocessing |Platform: All
Cpu: Unspecified |
--+-
Comment(by huhabla):
Replying to [comment:2 wenzeslaus]:
Thanks Soeren, this is what I hoped for and I'm looking forward how this
will continue. I just quickly looked at this topic for now. So, just some
small notes now.
I needed to do this changes to get it working:
{{{
#!diff
@@ -19,12 +19,13 @@
from multiprocessing import Process, Lock, Pipe
import logging
from ctypes import *
-from core import *
+from grass.script.core import *
import grass.lib.gis as libgis
import grass.lib.raster as libraster
import grass.lib.vector as libvector
import grass.lib.date as libdate
import grass.lib.raster3d as libraster3d
+from datetime import datetime
}}}
Thanks for the fix, the new version should work now in any directory.
I though I understand how it works with exceptions and doctest but
apparently not, even I tried verbose mode:
{{{
python -m doctest -v c_library_interface.py
}}}
Please try:
{{{
python c_library_interface.py -v
}}}
Anyway test passes. Would it be possible to write a test also for the
case of waiting, killing and restarting the server? I mean the case when I
run something with `call_no_wait(function=aaa)`, it runs for a long time,
meanwhile I call `call(function=bbb)` but then `bbb` fails and kills the
server. Does this make sense?
Yes it does, but i was not able to fix the deadlock that appeared when
mixing waiting calls with no waiting calls and a fatal error killed
the subprocess. So i removed the no wait calls from the server interface.
About the interface. I'm not sure about how to report errors. The
functions might need to accept keyword arguments in some way. How to deal
with objects, e.g. how to use pyGRASS through this?
In the latest implementation the subprocess creates an exception that is
send through the pipe to the server and is raised there.
If an object is picklable then it can be send between processes.
I was not thinking about how to deal with wrapping all C functions. But
maybe manual wrapping it is inevitable if we want really Python(ic)
interface. We already have pyGRASS for this reason.
The last not is about naming. Although it is a well established practice
in GRASS that things are named with two or more different names, I would
suggest to use only one name for newly created things. We are at the
beginning, we can always rename but it would be safer to use one name at
time. `*C*Interface` is a nice name (although it it might by also
`*Python*Interface`) but it is more for the particular implementation
(interface for temporal framework or to messages) than for the general
thing (server). `RPCServer` is very general, not sure about that. What
about `CFunctionCaller`? I also heard suggestion `NoGFatal_Huray!!!` from
someone here but that's not a valid Python identifier, although, with
unicode identifiers, there might be a way. No strong opinion here.
The latest RPC server incarnation does not care what kind of functions it
should call, hence it is very generic and takes care about killed
subprocesses. You can define any arbitrary function with arguments that
are picklable and pass it to the RPC server to call it in the subprocess.
But maybe it would be better to use sharedctypes memory in case of
digitizing rather than a pipe?
--
Ticket URL: http://trac.osgeo.org/grass/ticket/2134#comment:11
GRASS GIS http://grass.osgeo.org
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] new flag to interpolate from nearby raster cells for v.what.rast
Hi, Hamish wrote: I just added in the dev branches two new flags for v.what.rast. [...] The second flag changes the default containing-grid-cell method to a weighted avg. of the 4 nearest raster cell centers (IDW). The search radius is not very big [...] AFAIK, IDW takes as argument the radius which is fixed to 1 which is questionable. I'm not sure I follow what you mean, but the distance in the 4-way inverse distance weighting in the new v.what.rast -i interpolate flag is calculated for each of the closest four raster cells' center coord by running G_distance() vs. the vector point's position, and then the weighting in the weighted average for each of those four cells' data values is by 1/distance^2. It doesn't assume that the 4 surrounding cells are equally distant: weightsum = valweight = 0; weight[i] = 1.0 / (distance[i] * distance[i]); weightsum += weight[i]; valweight += weight[i] * nearby_d_val[i]; value = valweight / weightsum; // 0./0. == nan The other thing of note is that if any of the nearest four cells are null then it discards it from the weighted average, and proceeds with n-1 elements. I find this a preferable compromise between the null if-any-nulls and the fallback to nearest neighbor if-any-nulls methods below; YMMV. Since v.what.rast samples a raster value at a given location, why not use the standard raster sampling methods nearest: nearest neighbor linear: linear interpolation cubic: cubic convolution lanczos: lanczos filter linear_f: linear interpolation with fallback cubic_f: cubic convolution with fallback lanczos_f: lanczos filter with fallback which are used by r.proj and i.rectify? The concept is essentially the same: for a given location, estimate the corresponding raster value. what I've called IDW is similar to the bilinear method above, with the exception of how surrounding NULLs are handled, and nearest is similar to the default behavior of v.what.rast. If someone wants to take it up and expand the options to use Rast_interp_bicubic() co., I'm happy for it, but would like to preserve the null handling and speed. The main bug so far is if the print-only flag is used, it doesn't yet bypass the checks for if the vector map has a table, is in the current mapset (is alterable), and to remove the need for topology. regards, Hamish ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Implementation of the High Pass Filter Additive Fusion technique (i.fusion.hpf)
Nick Ves wrote: The HPF algorithm about the histrogram matching states Stretch the new multispectral image to match the mean and standard deviation of the original (input) multispectral image In that context why it is wrong to do: Ouput - output/sddev(output)*sddev(input) Output - Output - mean(output) + mean(input) Moritz Lennert: To give the 'new' image the same mean and stddev of the 'input' image: (new - mean(new)) / stddev(new) * stddev(input) + mean(input) Not entirely sure, I think it works. Taken a fragment from the publicly available QuickBird2 image over Sri Lanka (the extent shown at http://grasswiki.osgeo.org/wiki/File:RGB_04APR05050541-M2AS-00186011_01_P001.jpg), get mean and sd for r, g and b # red mean_red=321.375 sd_red=172.756 # hpf_red mean_red_hpf=321.364 sd_red_hpf=238.391 # green mean_green=434.033 sd_green=159.9 # hpf_green mean_green_hpf=434.02 sd_green=258.249 # blue mean_blue=285.168 sd_blue=75.0218 # hpf_blue mean_blue_hpf=285.161 sd_blue_hpf=145.395 and then r.mapcalc hpf_red_histomatched = ( hpf_red - $mean_red_hpf ) / $sd_red_hpf * $sd_red + $mean_red --o r.mapcalc hpf_blue_histomatched = ( hpf_blue - $mean_blue_hpf ) / $sd_blue_hpf * $sd_blue + $mean_blue --o r.mapcalc hpf_green_histomatched = ( hpf_green - $mean_green_hpf ) / $sd_green_hpf * $sd_green + $mean_green --o check output stats # r.univar hpf_red_histomatched mean: 321.375 standard deviation: 172.756 # r.univar hpf_green_histomatched mean: 434.033 standard deviation: 159.9 # r.univar hpf_blue_histomatched mean: 285.168 standard deviation: 75.0217 # reset colors to match originals doesn't play exactly nice, so r.colors hpf_red_histomatched color=grey -e # repeat for green, blue # draw d.rgb ... Looks nice! Is it that? Nikos ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
