[GRASS-dev] grass.jupyter Mini Project Update
Hi all, This is my last update before the final report! I'm finally wrapping up work on the grass.jupyter mini grant. We're just putting the final changes in the two PRs that I've been working on for a while [1, 2] and getting ready to implement the grass.jupyter naming conventions before the release of 8.2 [3]. Here's a quick summary: *1) What have I accomplished since my last update?* - I continue to have regular meetings with Vashek Petras. This week, we talked about testing and some minor edits for the existing pull requests [1,2] - Wrote tests for ReprojectionRenderer - a class that handles reprojecting vector and raster data for folium [2] - Had some productive discussions about naming conventions for grass.jupyter [3] *2) What do I plan to do next week?* - Merge existing pull requests [1, 2] - Implement changes to grass.jupyter class names [3] - Write final report *3) Am I blocked on anything?* - No Feedback, thoughts and comments welcome! Have a great week, Caitlin [2] https://github.com/OSGeo/grass/pull/2010 [3] https://github.com/OSGeo/grass/pull/2208 [4] https://github.com/OSGeo/grass/issues/2283 ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] multiprocessing problem
As I said, you can sum the values for each pixel so you don't store all the differences, that gets rid of the memory problem, but of course it will still be slow if it's not parallelized: vals = np.array([np.sum(np.abs(y - array.flat)) for y in array.flat]) Note that I didn't check thoroughly if the computation by itself is correct, i.e. you get the correct value in terms of the index definition. One other idea is to avoid some of the computations since you are in fact computing the distances twice (distance from pixel 1 to pixel 2 and vice versa). Also, do you actually need to compute this for the entire raster, shouldn't this be more a moving window approach, so you would restrict the distance computation only to a window around that pixel? Anna On Mon, Apr 11, 2022 at 2:09 AM Luca Delucchi wrote: > On Fri, 8 Apr 2022 at 16:33, Anna Petrášová wrote: > > > > Hi Luca, > > > > Hi Anna, > > > I would say the biggest problem is the memory, I tried to run it and it > consumes way too much memory. Maybe you could process the differences from > each pixel (compute the sum) as they are computed, not collect it and do it > in the end. Otherwise you can significantly speed it up simply with one > core by using numpy in a better way: > > > > vals = np.array([np.abs(y - array.flat) for y in array.flat]) > > ... > > out = np.sum(vals) / number2 > > > > yes, this work better then my solution, but increasing the number of > pixels I get the process killed. I have 16GB RAM and I was not able to > process 8 cells > I tried a few different solutions but the result is always the same. > > -- > ciao > Luca > > www.lucadelu.org > ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] multiprocessing problem
On Fri, 8 Apr 2022 at 16:33, Anna Petrášová wrote: > > Hi Luca, > Hi Anna, > I would say the biggest problem is the memory, I tried to run it and it > consumes way too much memory. Maybe you could process the differences from > each pixel (compute the sum) as they are computed, not collect it and do it > in the end. Otherwise you can significantly speed it up simply with one core > by using numpy in a better way: > > vals = np.array([np.abs(y - array.flat) for y in array.flat]) > ... > out = np.sum(vals) / number2 > yes, this work better then my solution, but increasing the number of pixels I get the process killed. I have 16GB RAM and I was not able to process 8 cells I tried a few different solutions but the result is always the same. -- ciao Luca www.lucadelu.org ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev
