Hi Dmitry again, on a different note. Since your assembly seems to be very expensive you would be a perfect candidate for multithreaded (i.e. shared memory parallelism) assembly. I implemented a prototype for the Box method on the branch feature/colored-smp-assembly (https://git.iws.uni-stuttgart.de/dumux-repositories/dumux/-/merge_requests/2640 <https://git.iws.uni-stuttgart.de/dumux-repositories/dumux/-/merge_requests/2640>). For this you need to install Intel’s thread building blocks library (TBB). Given that I implemented the colouring correctly, you should not get data races and for expensive assembly per element as you certainly have you should get a basically optimal speed-up when using more cores (even on smallish grids).
I’d be happy if you give it a try and give feedback if this worked. Best wishes, Timo > On 14. Oct 2021, at 11:39, Timo Koch <koch_t...@hotmail.com> wrote: > > > >> On 14. Oct 2021, at 00:12, Dmitry Pavlov <dmitry.pav...@outlook.com >> <mailto:dmitry.pav...@outlook.com>> wrote: >> >> Hello, >> >> I am making a compositional oil-gas simulator in DuMux. An important part of >> it, and arguably the most CPU-demanding, is the determination if the given >> composition is going to stay in one phase (stability test) and, in case the >> stability test returned false, the two-phase flash. >> >> VolumeVariables::update seems to be the most appropriate place to do this >> kind of calculations. Please correct me if I am wrong. >> >> I defined my own MyVolumeVariables and I call completeFluidState() in its >> update() method every time. Now, the less MyVolumeVariables::update() calls, >> the better. I found out that my program performs far more calls than I >> anticipated. >> >> 1. Some of the extra calls were coming from the (unmerged) fix >> <https://git.iws.uni-stuttgart.de/dumux-repositories/dumux/-/merge_requests/2146> >> that allowed relative permeabilities of an scv to depend of neighbor dofs. >> The fix was needed for another problem (surfactant simulation), so I >> reverted it for now. OK. >> >> 2. Some of the extra calls were coming from the PrimaryVariableSwitch >> machinery which is not required at all in the task. I set my own >> PrimaryVariableSwitch implementation via VolumeVariables with an empty >> update(). OK. >> >> > > Hi Dmitry, > > I would have to look into this a bit closer to give better answers but some > quick answers for now. > For 2. that’s the right thing to do. There is also the > NoPrimaryVariableSwitch which should do exactly that, i.e. nothing. >> 3. I am using the box method. Each node has its primary variables that go >> into MyVolumeVariables::update() and then into completeFluidState(). This is >> good. What is bad is that this is done independently for every scv that is >> connected to this node. So with a rectangular 2D grid I get four >> completeFluidState() calls with the same primary variables. So I am >> repeating the same calculations four times. Can this be avoided at present? >> Can this be avoided in principle? >> > Dumux chooses the most general case here (although arguably inconsistent this > is not done for the point 1. above as you already noticed). The volvars might > indeed be different for every scv at the node so this has to be done in the > general case. > On a rectangular 2D grid you should probably get 8 calls, no? 4 for the > residual and 4 for the deflected residual to compute the derivative. > Avoiding this is not possible out-of-the-box, but you can of course achieve > this by implementing a caching mechanism, e.g. in the GridVolumeVariables > (which then probably need to be accessible through the problem since that’s > what you get in volvars.update). > (also see below if you haven’t enabled the out-of-the-box caching already). >> 4. As soon as one step of the Newton method is completed, >> computeResidualReduction_ is called, then in calls assembleResidual, and >> after that, MyPrimaryVariables::update() for every scv, without any change >> in primary variables, is being called twise: first via >> >> bindLocalViews -> curElemVolVars.bind -> bindElement >> >> and then via >> >> bindLocalViews -> prevElemVolVars.bindElement >> >> I honestly do not understand what is going on here. >> > I’m not 100% sure if this is case here. But there is a caching mechanism > already implemented. So if volvar updates are expensive you definitely want > to use > template<class TypeTag> > struct EnableGridVolumeVariablesCache<TypeTag, TTag::YOURTYPETAG> { static > constexpr bool value = true; }; > template<class TypeTag> > struct EnableGridFluxVariablesCache<TypeTag, TTag::YOURTYPETAG> { static > constexpr bool value = true; }; > if you don’t have this enabled already? This will require more memory but > avoids unnecessary updates. > But usually for computing a residual criterion for the Newton you definitely > need to do at least one more update with the final solution. But using > caching you can avoid this then for the next newton step and also the > bindLocalViews -> prevElemVolVars.bindElement should be a loop. > >> 5. Then the next step is started, during which the system is assembled. >> There, MyPrimaryVariables::update() is called 7 times for each scv. I have 5 >> components, so 5 dofs (pressure + 4 mole fractions). Why 7 calls and not 5? >> I figure that 5 calls should be made to calculate the numerical derivatives >> w. r. t. the current state that has been already assembled in the end of the >> previous step. Even if we reassemble it, it is expected that we have only 6 >> calls. >> >> > > Sorry I also can’t answer that without looking more into it. > > Maybe some of the suggestions above already help? > > Best wishes, > Timo >> >> Please bear with me as I may have misconceptions about the box method and >> the DuMux ways of work. The above is not critique but a call for advice. >> >> >> >> Best regards, >> >> Dmitry >> >> >> >> _______________________________________________ >> DuMux mailing list >> DuMux@listserv.uni-stuttgart.de <mailto:DuMux@listserv.uni-stuttgart.de> >> https://listserv.uni-stuttgart.de/mailman/listinfo/dumux
_______________________________________________ DuMux mailing list DuMux@listserv.uni-stuttgart.de https://listserv.uni-stuttgart.de/mailman/listinfo/dumux
