hi Erik, i've made a gist here: https://gist.github.com/mzilhao/6eadbee1abd6ffb04faa1857e84d72a8 i've added the code snippet and (some of) the corresponding output from processor 2, as well as the parfile used. this parfile uses some private thorns, but i think the only relevant thing is the grid structure and coordinates. also, this used 48 processors. is there anything else i should provide?
thanks, Miguel On 25/06/19 19:59, Erik Schnetter wrote: > Miguel > > I do not know what you see, thus I cannot comment. Can you post your > output/results somewhere, e.g. on "gist.github.com"? > > -erik > > On Tue, Jun 25, 2019 at 2:53 PM Miguel Zilhão > <miguel.zilhao.nogue...@tecnico.ulisboa.pt> wrote: >> >> hi all, >> >> thanks for your replies. i will try to implement something based on these >> suggestions. in any case >> though, it still seems to me that the behaviour of the Sn function from the >> Interpolate2 thorn is >> not the expected one... >> >> i understand that some points can be both "outer boundary" and >> "inter-process boundary", but when >> using an outer radius of 128, i wouldn't expect a point at radius~60 to ever >> be flagged as "outer >> boundary"... yet, this does happen for some of the tests that i've >> performed. if this is indeed the >> expected behaviour, maybe some clarification could be added to its >> interface.ccl, or documentation? >> >> thanks, >> Miguel >> >> >> On 25/06/19 15:24, Erik Schnetter wrote: >>> Miguel >>> >>> Grid functions have edges and corners, and thus there are points that >>> are both "outer boundary" and "inter-process boundary". >>> >>> The canonical way to find out where the outer boundaries are is to call >>> >>> # The location of the boundary points >>> CCTK_INT FUNCTION MultiPatch_GetBoundarySpecification \ >>> (CCTK_INT IN map, \ >>> CCTK_INT IN size, \ >>> CCTK_INT OUT ARRAY nboundaryzones, \ >>> CCTK_INT OUT ARRAY is_internal, \ >>> CCTK_INT OUT ARRAY is_staggered, \ >>> CCTK_INT OUT ARRAY shiftout) >>> >>> This specifies, for a given map, how many boundary points there are >>> (nboundaryzones). >>> >>> You also call >>> >>> CCTK_INT FUNCTION \ >>> MultiPatch_GetBbox \ >>> (CCTK_POINTER_TO_CONST IN cctkGH, \ >>> CCTK_INT IN size, \ >>> CCTK_INT OUT ARRAY bbox) >>> >>> to find out whether each of the 6 boundaries are outer boundaries. >>> "Outer boundary" here means that you need to apply an outer boundary >>> condition. Note that this might include some ghost zones. Depending on >>> your setup, you can either synchronize before applying outer boundary >>> conditions, and then apply them on ghost zones as well, or you can >>> skip applying outer boundary conditions on outer boundary points that >>> are ghost points, and then synchronize later. >>> >>> To find out which points are ghost points, look at cctk_bbox in cctkGH. >>> >>> -erik >>> >>> >>> On Tue, Jun 25, 2019 at 9:50 AM Haas, Roland <rh...@illinois.edu> wrote: >>>> >>>> Hello Miguel, >>>> >>>> the official way would be to use CoordBase's GetBoundarySizesAndTypes >>>> function or replicate the code in repos/mclachlan/ML_BSSN/src/Kranc.cc, >>>> specifically GetBoundaryInfo. >>>> >>>> The repsonses in the is_symbnd from GetBoundarySizesAndTypes and >>>> GetBoundaryInfo likely differ since the former does not know about Llama >>>> so will classify Llama's inter-patch boundaries as "symmetry" boundaries >>>> while the latter does not do so. >>>> >>>> Their answer's for physical boundaries (which may not quite be the same >>>> as "outer" boundaries if symmetry boundaries count as "outer") should >>>> be identical though. And you are looking for the physical boundaries >>>> after all. >>>> >>>> Yours, >>>> Roland >>>> >>>>> hi Roland, all, >>>>> >>>>> i'm coming back to this point since i'm now realizing that my >>>>> interpretation was not correct... just to recall, i'm imposing boundary >>>>> conditions in the following fashion: >>>>> >>>>> reflevel = GetRefinementLevel(cctkGH) >>>>> map = MultiPatch_GetMap(cctkGH) >>>>> if (reflevel /= 0 .or. map == 0) return >>>>> >>>>> do j = 1, cctk_lsh(2) >>>>> do i = 1, cctk_lsh(1) >>>>> do k = cctk_lsh(3)-cctk_nghostzones(3)+1, cctk_lsh(3) >>>>> if (Sn(i,j,k) == -2) then >>>>> >>>>> [my BCs go here] >>>>> >>>>> end if >>>>> end do >>>>> end do >>>>> end do >>>>> >>>>> and my assumption was that the line "if (Sn(i,j,k) == -2)" would >>>>> guarantee that this would only run in the physical boundary region, as >>>>> per the comment in the file interface.ccl from llama/Interpolate2: >>>>> >>>>> CCTK_INT source_patch TYPE=gf TAGS='Checkpoint="no" >>>>> Prolongation="none"' >>>>> { >>>>> Sn >>>>> } "source patch number, -1 for interior points, -2 for outer >>>>> boundary points, -3 for inter- >>>>> processor ghost points" >>>>> >>>>> this interpretation was consistent with what it's done in the thorn >>>>> LlamaWaveToy (where the BC are imposed in a similar way). >>>>> >>>>> however, when running with several processors, it seems that there are >>>>> some inter-processor points marked with Sn = -2, and therefore the loop >>>>> above is called outside my physical boundary region (i noticed this by >>>>> adding write statements inside the loop above). >>>>> >>>>> so i have the following questions: >>>>> >>>>> - what exactly does "outer boundary points" mean for the Sn function? >>>>> is there a bug, or did i just misinterpreted that "outer boundary" means >>>>> the "physical" boundary? >>>>> >>>>> - what would be the canonical way of checking whether a point belongs >>>>> to the physical outer boundary (ie, where BCs should be specified) when >>>>> using Llama? >>>>> >>>>> thanks, >>>>> Miguel >>>>> >>>>> >>>>> On 18/06/19 16:00, Haas, Roland wrote: >>>>>> Hello Miguel, >>>>>> >>>>>>> oh, but won't this be handled by the >>>>>>> >>>>>>> if (Sn(i,j,k) == -2) >>>>>>> >>>>>>> line above? >>>>>> I see. I had no idea that this is what the grid function does. >>>>>> >>>>>>> from the description of the Sn function in the Interpolate2 thorn: >>>>>>> >>>>>>> CCTK_INT source_patch TYPE=gf TAGS='Checkpoint="no" >>>>>>> Prolongation="none"' { >>>>>>> Sn >>>>>>> } "source patch number, -1 for interior points, -2 for outer >>>>>>> boundary points, -3 for inter- processor ghost points" >>>>>>> >>>>>>> and also from the way this is done in LlamaWaveToy, i was assuming >>>>>>> that Sn(i,j,k) == -2 would guarantee that the points belong to the >>>>>>> outer boundary (ie, points where i want to specify a physical >>>>>>> boundary condition). is this assumption not correct? >>>>>> Yes, in that case this should work fine. In fact it is probably more >>>>>> correct then what I did b/c due to the way Llama works may suggestion >>>>>> would likely have treated the inter-patch bondaries in Llama as outer >>>>>> boundaries (since they are "symmetry" boundaries to Cactus). >>>>>> >>>>>> So you should be fine actually. >>>>>> >>>>>> Yours, >>>>>> Roland >>>>>> >>>> >>>> >>>> >>>> -- >>>> My email is as private as my paper mail. I therefore support encrypting >>>> and signing email messages. Get my PGP key from http://pgp.mit.edu . >>>> _______________________________________________ >>>> Users mailing list >>>> Users@einsteintoolkit.org >>>> http://lists.einsteintoolkit.org/mailman/listinfo/users >>> >>> >>> > > > _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users