#3166: Parallelization with tiling for grass.script ------------------------------+------------------------- Reporter: wenzeslaus | Owner: grass-dev@… Type: enhancement | Status: new Priority: normal | Milestone: 7.4.0 Component: Python | Version: unspecified Keywords: script, parallel | CPU: Unspecified Platform: Unspecified | ------------------------------+------------------------- At the same time as r69507, I was working on a simpler approach based on `grass.script`. At this point, it can do tiling and patching of rasters and partially 3D rasters. Series of commands is executed on each tile. This and also the patching runs in parallel. The syntax is very similar to `grass.script` with a convenient function for tile naming (which is partially a user responsibility).
Here is an tiling example: {{{ #!python # this is the control object tiled_workflow = TiledWorkflow(nprocs=4, width=500, height=500, overlap=10) for namer, workflow in tiled_workflow: slope = namer.name('raster', 'slope') aspect = namer.name('raster','aspect') # now do all as usually, workflow is equivalent of `grass.script` workflow.run_command('r.slope.aspect', elevation='fractal_surface', slope=slope) workflow.run_command('r.slope.aspect', elevation='fractal_surface', aspect=aspect) workflow.parse_command('g.region', flags='pg') # nothing was actually done till now # do the parallel processing and patching results = tiled_workflow.execute() # iterate over the results (here from g.region) for result in results: for key, value in result.iteritems(): print key, value }}} Example using much smaller portion of the API. Creates list of modules which are then executed in the background. When import of the `parallel` module fails, `grass.script` is used instead without any changed in the main part of the code. {{{ #!python try: from grass.script.parallel import ModuleCallList, execute_by_module call = ModuleCallList() parallel = True except ImportError: call = gs # fall back to grass.script parallel = False for i in range(map_min, map_max + 1): call.mapcalc(expr, num=i) if parallel: execute_by_module(call, nprocs=4) }}} The current code uses `r.mapcalc` for patching and PyGRASS code computing the tiles. One of the main issues with the current code is that it does not finish when there is an error in the executed module. -- Ticket URL: <https://trac.osgeo.org/grass/ticket/3166> GRASS GIS <https://grass.osgeo.org> _______________________________________________ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev