Re: [GRASS-dev] multiprocessing problem

[Luca Delucchi]

On Fri, 8 Apr 2022 at 11:17, Stefan Blumentrath
 wrote:
>
> Ciao Luca,
>

Ciao Stefan

> Yes, you could also consider looping over e.g. rows (maybe in combination 
> with "np.apply_along_axis") so you could put results easier back together to 
> a map if needed at a later stage.
>
> In addition, since you use multiprocessing.Manager, you may try to use 
> multiprocessing.Array: 
> https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Array
>
> E.g. here:
> https://github.com/lucadelu/grass-addons/blob/5ca56bdb8b3394ebeed23aa5b3240bf6690e51bf/src/raster/r.raoq.area/r.raoq.area.py#L81
>
> According to the post here: 
> https://medium.com/analytics-vidhya/using-numpy-efficiently-between-processes-1bee17dcb01
> multiprocessing.Array is needed to put the numpy array into shared memory and 
> avoid pickling.
>
> I have not tried or investigated myself, but maybe worth a try...
>

Yes I saw it but I didn't try before. I tried last days but I didn't
get any improvements, I will try in the coming days

> Cheers
> Stefan
>

-- 
ciao
Luca

www.lucadelu.org
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] multiprocessing problem

[Luca Delucchi]

On Fri, 8 Apr 2022 at 16:33, Anna Petrášová  wrote:
>
> Hi Luca,
>

Hi Anna,

> I would say the biggest problem is the memory, I tried to run it and it 
> consumes way too much memory. Maybe you could process the differences from 
> each pixel (compute the sum) as they are computed, not collect it and do it 
> in the end. Otherwise you can significantly speed it up simply with one core 
> by using numpy in a better way:
>
> vals = np.array([np.abs(y - array.flat) for y in array.flat])
> ...
> out = np.sum(vals) / number2
>

yes, this work better then my solution, but increasing the number of
pixels I get the process killed. I have 16GB RAM and I was not able to
process 8 cells
I tried a few different solutions but the result is always the same.

-- 
ciao
Luca

www.lucadelu.org
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev