On Thu, Feb 18, 2021 at 4:04 PM Fande Kong <fdkong...@gmail.com> wrote:
> > > On Thu, Feb 18, 2021 at 1:55 PM Junchao Zhang <junchao.zh...@gmail.com> > wrote: > >> VecScatter (i.e., SF, the two are the same thing) setup (building various >> index lists, rank lists) is done on the CPU. is1, is2 must be host data. >> > > Just out of curiosity, is1 and is2 can not be created on a GPU device in > the first place? That being said, it is technically impossible? Or we just > did not implement them yet? > Simply because we do not have an ISCUDA class. > > Fande, > > >> When the SF is used to communicate device data, indices are copied to the >> device.. >> >> --Junchao Zhang >> >> >> On Thu, Feb 18, 2021 at 11:50 AM Patrick Sanan <patrick.sa...@gmail.com> >> wrote: >> >>> I'm trying to understand how VecScatters work with GPU-native Kokkos >>> Vecs. >>> >>> Specifically, I'm interested in what will happen in code like in >>> src/vec/vec/tests/ex22.c, >>> >>> ierr = VecScatterCreate(x,is1,y,is2,&ctx);CHKERRQ(ierr); >>> >>> (from >>> https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44 >>> ) >>> >>> Here, x and y can be set to type KOKKOS using -vec_type kokkos at the >>> command line. But is1 and is2 are (I think), always >>> CPU/host data. Assuming that the scatter itself can happen on the GPU, >>> the indices must make it to the device somehow - are they copied there when >>> the scatter is created? Is there a way to create the scatter using indices >>> already on the GPU (Maybe using SF more directly)? >>> >>>