#3166: Parallelization with tiling for grass.script --------------------------+------------------------------ Reporter: wenzeslaus | Owner: grass-dev@… Type: enhancement | Status: new Priority: normal | Milestone: 7.4.0 Component: Python | Version: unspecified Resolution: | Keywords: script, parallel CPU: Unspecified | Platform: Unspecified --------------------------+------------------------------
Comment (by wenzeslaus): Replying to [comment:1 mlennert]: > Wow, this looks great ! Note also what is in PyGRASS. Especially after r69507. > Could you just explain the relation Partial duplication. As in the case of `grass.script` and `grass.pygrass.modules`. > difference between this and the GridModule in pygrass ? * This supports list of modules (workflow) which are executed subsequently on the given tile. * User needs to prepare the list in a for loop (as opposed to not using any for loop). This is because it is in fact derived from the non-tiled parallelization API which is more general, so the user in fact loops over what needs to be done (with or without parallelization). * Related to that, the API for simple parallel processing, parallel processing of series of maps, and tiled parallel processing of series of maps is the same. * User needs to "help" the functions and objects by providing the with some metadata, i.e. types and names of the maps to patch (for patching), because no module run metadata are available in `grass.script`. * Some of the execution details are lost, e.g. only last command's textual output is preserved. * `GridModule` uses separate mapsets for individual tiles, this uses `WIND_OVERRIDE`. * `GridModule` uses PyGRASS ctypes wrappers for patching, this (potentially huge) expression `r.mapcalc` and `r3.mapcalc`. * The interface is like `grass.script`, not like `grass.pygrass.modules`. * It is not complete. * The implementation is now 300 lines. `MultiModule` alone has 200. * It uses from multiprocessing `Pool.map_async` function (which may be the cause of problem with interrupting). -- Ticket URL: <https://trac.osgeo.org/grass/ticket/3166#comment:3> GRASS GIS <https://grass.osgeo.org> _______________________________________________ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev