Soeren wrote: > The computation time of overlapping regions should be the > same in python as in C. > IMHO it is easier to program a python module which computes > the tiles, split up the vector points based on the tiles and > distribute the data across a grid or cloud for computation. > After the distributed processes are finished, the script can > gather the data and merge it together. > > If you implement this into the lidar modules, each module > needs to be changed to support MPI. Using MPI is IMHO much more > complex, then just running the modules on different datasets > on different nodes, gathering and patching the data. ... > Maybe the overlapping algorithm can be simplified in the > python version? > In my vision, the LIDAR computation could be run in an > elastic computation cloud (EC2) as WPS backend using > recursive tiling algorithm for data distribution.
thinking big! for my 2c (fwiw I run about 160 computational-hours per day split up and dynamically sent out to a couple of clusters using MPI), I would first ask how much processing are we talking about? If the typical large-case lidar tools job only took ~3 hours to run on a modern PC, I'd much prefer the simplicity of OpenMP and run it on the same quad-core desktop or 8-core Xeon server that the GIS is running on. If the jobs could take 12-24 hours and have to happen often I would more seriously consider using MPI. Setting up the mpd daemon and organizing the mpirun jobs is a lot of overhead, which for smaller jobs seems slightly counter-productive to me. also, I think it is quite natural to have the control and management scripts in python, but get worried about splitting up low-level processing into a hybrid python/C mix. for one thing it makes the code harder to follow/understand, for another you are at risk from different component versions interacting in different ways. (see wxNviz, vdigit problems on WinGrass..) (and of course this has to be an optional extra, and I'm all for exploring all these different methods .. we don't know which one will win out/survive in a few years time when 16-core workstations are the norm) regards, Hamish _______________________________________________ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev