On Fri, Jun 17, 2022 at 11:02 AM Barry Smith <bsm...@petsc.dev> wrote:
>
> You do not need the loop over size. Each rank sets options and options
> prefix for its local objects and never anyone elses.
>
> char prefix[10];
>
> sprintf(prefix,"sub_%d_",rank);
>
> SNESNASMGetSNES(snes,0,&subsnes);
>
> if (rank <size-1) {
> SNESSetType(subsnes,SNESNEWTONLS); CHKERRQ(ierr); // newton
> for regular domains
> } else {
> SNESSetType(subsnes,SNESFAS); CHKERRQ(ierr); // fas for last
> domain
> }
> SNESSetOptionsPrefix(subsnes,prefix);
> }
> }
>
>
> To get the prefix to work try calling SNESSetFromOptions(subsnes);
> immediately after your SNESSetOptionsPrefix(subsnes,prefix); call
>
> Matt, it looks like there may be a bug in NASM, except in one
> particular case, it never calls SNESSetFromOptions() on the subsenses.
>
I bet Barry is correct. I can fix it, but unfortunately, I leave for a week
long conference tomorrow, of which I am an organizer, so I don't
think I can do it for a week.
Thanks,
Matt
> Barry
>
>
> On Jun 17, 2022, at 10:47 AM, Takahashi, Tadanaga <t...@njit.edu> wrote:
>
> Thank you. I am now able to pull each subsnes, change its snes type
> through the API, and set a prefix. This is my updated code:
>
> SNES snes, subsnes;
> PetscMPIInt rank, size;
> ...
> ierr = SNESCreate(PETSC_COMM_WORLD,&snes); CHKERRQ(ierr);
> ierr = SNESSetType(snes,SNESNASM); CHKERRQ(ierr);
> ierr = SNESNASMSetType(snes,PC_ASM_RESTRICT); CHKERRQ(ierr);
> ...
> ierr = SNESSetFromOptions(snes); CHKERRQ(ierr);
> ierr = SNESSetUp(snes); CHKERRQ(ierr);
> PetscPrintf(PETSC_COMM_WORLD, "Size = %d\n",size);
> PetscBarrier(NULL);
> for (i=0; i<size; i++) {
> char prefix[10];
> sprintf(prefix,"sub_%d_",i);
> if(i==rank) {
> ierr = SNESNASMGetNumber(snes,&Nd);
> printf("rank = %d has %d block(s)\n",i,Nd);
> if (i <size-1) {
> SNESNASMGetSNES(snes,0,&subsnes);
> SNESSetType(subsnes,SNESNEWTONLS); CHKERRQ(ierr); // newton
> for regular domains
> } else {
> SNESNASMGetSNES(snes,0,&subsnes);
> SNESSetType(subsnes,SNESFAS); CHKERRQ(ierr); // fas for last
> domain
> }
> SNESSetOptionsPrefix(subsnes,prefix);
> }
> }
> ierr = SNESSetFromOptions(snes); CHKERRQ(ierr);
> ...
> ierr = SNESSolve(snes,NULL,u_initial); CHKERRQ(ierr);
>
> However, I still cannot change SNES, KSP, and PC types for
> individual domains through the command arguments. I checked the subdomains
> with -snes_view ::ascii_info_detail and it does show that the prefixes
> are properly changed. It also shows that the SNES type for the last domain
> was successfully changed. But for some reason, I only have access to the
> SNES viewer options during runtime. For example, if I run mpiexec -n 4
> ./test1 -sub_0_ksp_type gmres -help | grep sub_0 I get the output:
>
> Viewer (-sub_0_snes_convergence_estimate) options:
> -sub_0_snes_convergence_estimate ascii[:[filename][:[format][:append]]]:
> Prints object to stdout or ASCII file (PetscOptionsGetViewer)
> -sub_0_snes_convergence_estimate
> binary[:[filename][:[format][:append]]]: Saves object to a binary file
> (PetscOptionsGetViewer)
> -sub_0_snes_convergence_estimate draw[:[drawtype][:filename|format]]
> Draws object (PetscOptionsGetViewer)
> -sub_0_snes_convergence_estimate socket[:port]: Pushes object to a Unix
> socket (PetscOptionsGetViewer)
> -sub_0_snes_convergence_estimate saws[:communicatorname]: Publishes
> object to SAWs (PetscOptionsGetViewer)
> Viewer (-sub_0_snes_view_pre) options:
> -sub_0_snes_view_pre ascii[:[filename][:[format][:append]]]: Prints
> object to stdout or ASCII file (PetscOptionsGetViewer)
> -sub_0_snes_view_pre binary[:[filename][:[format][:append]]]: Saves
> object to a binary file (PetscOptionsGetViewer)
> -sub_0_snes_view_pre draw[:[drawtype][:filename|format]] Draws object
> (PetscOptionsGetViewer)
> -sub_0_snes_view_pre socket[:port]: Pushes object to a Unix socket
> (PetscOptionsGetViewer)
> -sub_0_snes_view_pre saws[:communicatorname]: Publishes object to SAWs
> (PetscOptionsGetViewer)
> Viewer (-sub_0_snes_test_jacobian_view) options:
> -sub_0_snes_test_jacobian_view ascii[:[filename][:[format][:append]]]:
> Prints object to stdout or ASCII file (PetscOptionsGetViewer)
> -sub_0_snes_test_jacobian_view binary[:[filename][:[format][:append]]]:
> Saves object to a binary file (PetscOptionsGetViewer)
> -sub_0_snes_test_jacobian_view draw[:[drawtype][:filename|format]] Draws
> object (PetscOptionsGetViewer)
> -sub_0_snes_test_jacobian_view socket[:port]: Pushes object to a Unix
> socket (PetscOptionsGetViewer)
> -sub_0_snes_test_jacobian_view saws[:communicatorname]: Publishes object
> to SAWs (PetscOptionsGetViewer)
> Viewer (-sub_0_snes_test_jacobian_display) options:
> -sub_0_snes_test_jacobian_display
> ascii[:[filename][:[format][:append]]]: Prints object to stdout or ASCII
> file (PetscOptionsGetViewer)
> -sub_0_snes_test_jacobian_display
> binary[:[filename][:[format][:append]]]: Saves object to a binary file
> (PetscOptionsGetViewer)
> -sub_0_snes_test_jacobian_display draw[:[drawtype][:filename|format]]
> Draws object (PetscOptionsGetViewer)
> -sub_0_snes_test_jacobian_display socket[:port]: Pushes object to a Unix
> socket (PetscOptionsGetViewer)
> -sub_0_snes_test_jacobian_display saws[:communicatorname]: Publishes
> object to SAWs (PetscOptionsGetViewer)
> Viewer (-sub_0_ksp_converged_reason) options:
> -sub_0_ksp_converged_reason ascii[:[filename][:[format][:append]]]:
> Prints object to stdout or ASCII file (PetscOptionsGetViewer)
> -sub_0_ksp_converged_reason binary[:[filename][:[format][:append]]]:
> Saves object to a binary file (PetscOptionsGetViewer)
> -sub_0_ksp_converged_reason draw[:[drawtype][:filename|format]] Draws
> object (PetscOptionsGetViewer)
> -sub_0_ksp_converged_reason socket[:port]: Pushes object to a Unix
> socket (PetscOptionsGetViewer)
> -sub_0_ksp_converged_reason saws[:communicatorname]: Publishes object to
> SAWs (PetscOptionsGetViewer)
> Viewer (-sub_0_snes_converged_reason) options:
> -sub_0_snes_converged_reason ascii[:[filename][:[format][:append]]]:
> Prints object to stdout or ASCII file (PetscOptionsGetViewer)
> -sub_0_snes_converged_reason binary[:[filename][:[format][:append]]]:
> Saves object to a binary file (PetscOptionsGetViewer)
> -sub_0_snes_converged_reason draw[:[drawtype][:filename|format]] Draws
> object (PetscOptionsGetViewer)
> -sub_0_snes_converged_reason socket[:port]: Pushes object to a Unix
> socket (PetscOptionsGetViewer)
> -sub_0_snes_converged_reason saws[:communicatorname]: Publishes object
> to SAWs (PetscOptionsGetViewer)
> Viewer (-sub_0_snes_view) options:
> -sub_0_snes_view ascii[:[filename][:[format][:append]]]: Prints object
> to stdout or ASCII file (PetscOptionsGetViewer)
> -sub_0_snes_view binary[:[filename][:[format][:append]]]: Saves object
> to a binary file (PetscOptionsGetViewer)
> -sub_0_snes_view draw[:[drawtype][:filename|format]] Draws object
> (PetscOptionsGetViewer)
> -sub_0_snes_view socket[:port]: Pushes object to a Unix socket
> (PetscOptionsGetViewer)
> -sub_0_snes_view saws[:communicatorname]: Publishes object to SAWs
> (PetscOptionsGetViewer)
> Viewer (-sub_0_snes_view_solution) options:
> -sub_0_snes_view_solution ascii[:[filename][:[format][:append]]]: Prints
> object to stdout or ASCII file (PetscOptionsGetViewer)
> -sub_0_snes_view_solution binary[:[filename][:[format][:append]]]: Saves
> object to a binary file (PetscOptionsGetViewer)
> -sub_0_snes_view_solution draw[:[drawtype][:filename|format]] Draws
> object (PetscOptionsGetViewer)
> -sub_0_snes_view_solution socket[:port]: Pushes object to a Unix socket
> (PetscOptionsGetViewer)
> -sub_0_snes_view_solution saws[:communicatorname]: Publishes object to
> SAWs (PetscOptionsGetViewer)
> Option left: name:-sub_0_ksp_type value: gmres
>
>
> Do you know what could be causing this?
>
> On Fri, Jun 17, 2022 at 10:00 AM Barry Smith <bsm...@petsc.dev> wrote:
>
>>
>> MPI_Comm_size(PETSC_COMM_WORLD,&size);
>> MPI_Comm_rank(PETSC_COMM_WORLD,&rank);
>>
>> SNESNASMGetSNES(snes,0,&subsnes);
>>> char prefix[10];
>>> sprintf(prefix,"sub_%d_",rank);
>>> SNESSetOptionsPrefix(subsnes,prefix);
>>>
>>
>>
>>
>> On Jun 17, 2022, at 9:35 AM, Matthew Knepley <knep...@gmail.com> wrote:
>>
>> On Fri, Jun 17, 2022 at 9:22 AM Takahashi, Tadanaga <t...@njit.edu>
>> wrote:
>>
>>> I'm having some trouble pulling out the subsolver. I tried to use
>>> SNESNASMGetSNES in a loop over each subdomain. However I get an error when
>>> I run the code with more than one MPI processors. Here is a snippet from my
>>> code:
>>>
>>> SNES snes, subsnes;
>>> PetscMPIInt rank, size;
>>> ...
>>> ierr = SNESCreate(PETSC_COMM_WORLD,&snes); CHKERRQ(ierr);
>>> ierr = SNESSetType(snes,SNESNASM); CHKERRQ(ierr);
>>> ierr = SNESNASMSetType(snes,PC_ASM_RESTRICT); CHKERRQ(ierr);
>>> ...
>>> ierr = SNESSetFromOptions(snes); CHKERRQ(ierr);
>>> ierr = SNESSetUp(snes); CHKERRQ(ierr);
>>> PetscPrintf(PETSC_COMM_WORLD, "Size = %d\n",size);
>>> for (i=0; i<size; i++) {
>>> PetscPrintf(PETSC_COMM_WORLD, "rank = %d\n",i);
>>> SNESNASMGetSNES(snes,i,&subsnes);
>>> // char prefix[10];
>>> // sprintf(prefix,"sub_%d_",i);
>>> // SNESSetOptionsPrefix(subsnes,prefix);
>>> }
>>> ...
>>> ierr = SNESSolve(snes,NULL,u_initial); CHKERRQ(ierr);
>>>
>>>
>>> And, here is the output of the code when I run with 2 MPI procs:
>>>
>>
>> SNESNASMGetSNES() gets the local subsolvers. It seems you only have one
>> per process.
>> You can check
>> https://petsc.org/main/docs/manualpages/SNES/SNESNASMGetNumber/
>>
>> Notice that your current code will not work because, according to your
>> explanation, you only want to change
>> the prefix on a single rank, so you need to check the rank when you do it.
>>
>> Thanks,
>>
>> Matt
>>
>>
>>> takahashi@ubuntu:~/Desktop/MA-DDM/C/Rectangle$ mpiexec -n 2 ./test1
>>> Size = 2
>>> rank = 0
>>> rank = 1
>>> [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [0]PETSC ERROR: Argument out of range
>>> [0]PETSC ERROR: No such subsolver
>>> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.17.1, unknown
>>> [0]PETSC ERROR: ./test1 on a linux-gnu-c-debug named ubuntu by takahashi
>>> Fri Jun 17 06:06:38 2022
>>> [0]PETSC ERROR: Configure options --with-mpi-dir=/usr --with-fc=0
>>> [0]PETSC ERROR: #1 SNESNASMGetSNES() at
>>> /home/takahashi/Desktop/petsc/src/snes/impls/nasm/nasm.c:923
>>>
>>>
>>> ===================================================================================
>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
>>> = RANK 0 PID 976566 RUNNING AT ubuntu
>>> = KILLED BY SIGNAL: 9 (Killed)
>>>
>>> ===================================================================================
>>>
>>> This error doesn't occur when I run this without MPI. However, I tried
>>> to change the prefix of the subdomain to `sub_0_` but I am not able to
>>> change the snes_type using this prefix. Running ./test1 -snes_view
>>> -help | grep sub_0_snes_type prints nothing.
>>>
>>> On Thu, Jun 16, 2022 at 6:23 PM Matthew Knepley <knep...@gmail.com>
>>> wrote:
>>>
>>>> On Thu, Jun 16, 2022 at 5:57 PM tt73 <t...@njit.edu> wrote:
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am using NASM as the outer solver for a nonlinear problem. For one
>>>>> of the subdomains, I want to run the local solve with a different set of
>>>>> options form the others. Is there any way to set options for each
>>>>> subdomain?
>>>>>
>>>>
>>>> I can see two ways:
>>>>
>>>> 1) Pull out the subsolver and set it using the API
>>>>
>>>> 2) Pull out the subsolver and give it a different prefix
>>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>>
>>
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
