Re: [Opm] OPM Flow multi-node simulations stuck at domain decomposition step

2020-03-11 Thread Kai Bao
Hi, Yogi, I see you are trying to run Norne with 144 processes. Did you see the problem with much less processes, for example 4 or 8 processes? In my opinion, with the current approach for domain decomposition, it can be challenging to run Norne with so many processes, considering the relative

[Opm] Initialization for steady state upscaling

2020-03-11 Thread Lezin Galibert
Hello everyone, I am trying to run steadystate_test_upscaling (from opm-upscaling) on a primary drainage case, and I am struggling to get convincing results. No matter what capillary number I use, the output relperms stay the same (down to the last digit). We are suspecting it might have to do wit

Re: [Opm] OPM Flow multi-node simulations stuck at domain decomposition step

2020-03-11 Thread Markus Blatt
On Tue, Mar 10, 2020 at 02:15:19PM -0700, Yogi Pandey wrote: > All, > I am trying to run OPM Flow simulations on multiple nodes. I have built OPM > Flow from source on Oracle Linux 7 OS (binary compatible with RHEL) with: > > [...] > > .OPM Flow modules are built using following commads:

Re: [Opm] OPM Flow multi-node simulations stuck at domain decomposition step

2020-03-11 Thread Markus Blatt
Hi Yogi, On Tue, Mar 10, 2020 at 02:15:19PM -0700, Yogi Pandey wrote: > Simulation is being run on 4 nodes with 32 processors each using following > command: > > mpirun --display-map -mca btl self -x UCX_TLS=rc,self,sm -x > HCOLL_ENABLE_MCAST_ALL=0 -mca coll_hcoll_enable 0 -x UCX_IB_TRAFFIC_CLA

Re: [Opm] OPM Flow multi-node simulations stuck at domain decomposition step

2020-03-11 Thread Atgeirr Rasmussen
Hi Yogi, Lots of things have changed in Flow for the initialization in parallel recently. Could you try to check out the previous release (2019.10) and check if you see the same problems? Atgeirr Frå: Opm på vegne av Markus Blatt Sendt: onsdag 11. mars 2020 1

[Opm] 2020.04 release

2020-03-11 Thread Joakim Hove
Dear opm community; we are approaching April and the 2020.04 release of opm/flow is soon imminent, and it is time to start planning the release process. If someone has planned to be the release manager for the 2020.04 release that is cool, otherwise opm-op can be responsible for managing the 2020.

Re: [Opm] OPM Flow multi-node simulations stuck at domain decomposition step

2020-03-11 Thread Yogi Pandey
Hi Kai, Thank you for your reply. I have run the Norne model with 1, 2, 4, 8, 16, 36, 72, and 108 processes successfully. The run gets stuck at 144 processes. As I mentioned, I did try a larger model, but even there I was not able to scale on more than 1 node for a certain configuration. Fin

Re: [Opm] OPM Flow multi-node simulations stuck at domain decomposition step

2020-03-11 Thread Yogi Pandey
Hi Dr. Blatt, I am using OPM Flow version 2020.04-pre. Each node has 384 GB memory, and I don't see any memory problem indicated by kswapd appearing. With some hit and trial I arrived at a model with just 1 producer and 1 injector, with 11 million cells, which scales on 4 nodes. My impression i

Re: [Opm] OPM Flow multi-node simulations stuck at domain decomposition step

2020-03-11 Thread Yogi Pandey
Dr. Blatt, That's a typo. Each node has 36 processors. Thank you, Yogi -Original Message- From: Markus Blatt [mailto:mar...@dr-blatt.de] Sent: Wednesday, March 11, 2020 5:09 AM To: opm@opm-project.org Subject: Re: [Opm] OPM Flow multi-node simulations stuck at domain decomposition step

Re: [Opm] OPM Flow multi-node simulations stuck at domain decomposition step

2020-03-11 Thread Yogi Pandey
Thank you for the suggestion Atgeirr. I'll try out the previous release 2019.10. - Yogi -Original Message- From: Atgeirr Rasmussen [mailto:atgeirr.rasmus...@sintef.no] Sent: Wednesday, March 11, 2020 6:46 AM To: opm@opm-project.org Subject: Re: [Opm] OPM Flow multi-node simulations stuck