Re: [petsc-users] Data types for local ID's and global ID's for large problems

[Jed Brown via petsc-users]

"Weston, Brian Thomas via petsc-users"  writes:

> Matt,
>
> Thanks for the quick reply. Users of our ALE3D hydro code at LLNL run very 
> large problems , which I believe can pass in large amounts of local integer 
> data to solver packages like HYPRE and PETSc for global solves. All of our 
> local integer data is stored with 32-bit indices and switching to 64-bit 
> indices would increase the memory footprint and data motion. This is a 
> problem when running on our BGQ machines like Sequoia machine and potentially 
> for our GPU machines.
>
> Is this not a performance issue for most PETSc users who are running very 
> large problems on 64-bit indices? And is there any interest in the future for 
> allowing local ID’s to be in 32-bit and global ID’s be in 64-bit?

The reason there is only one PetscInt is to reduce the number of
combinations to be tested (since C has weak typedefs).  The performance
impact observed in practice tends to be quite small.  PETSc might
consider internally changing the storage for common matrix types to use
a separate typedef so that PetscInt would be 64-bit but "PetscMatInt"
could be 32-bit, but I don't think anyone wants that to escape into the
public interface so it wouldn't have any impact on your code, just a
(slight) optimization --with-64-bit-indices.  (Note that some users need
64-bit on a single rank so it needs to be possible to use 64-bit
PetscMatInt.)

Re: [petsc-users] Data types for local ID's and global ID's for large problems

[Weston, Brian Thomas via petsc-users]

Matt,

Thanks for the quick reply. Users of our ALE3D hydro code at LLNL run very 
large problems , which I believe can pass in large amounts of local integer 
data to solver packages like HYPRE and PETSc for global solves. All of our 
local integer data is stored with 32-bit indices and switching to 64-bit 
indices would increase the memory footprint and data motion. This is a problem 
when running on our BGQ machines like Sequoia machine and potentially for our 
GPU machines.

Is this not a performance issue for most PETSc users who are running very large 
problems on 64-bit indices? And is there any interest in the future for 
allowing local ID’s to be in 32-bit and global ID’s be in 64-bit?

Best,
Brian

From: Matthew Knepley 
Date: Wednesday, January 9, 2019 at 2:56 PM
To: "Weston, Brian Thomas" 
Cc: PETSc 
Subject: Re: [petsc-users] Data types for local ID's and global ID's for large 
problems

On Wed, Jan 9, 2019 at 5:53 PM Weston, Brian Thomas via petsc-users 
mailto:petsc-users@mcs.anl.gov>> wrote:
Does PETSc have different data types for local ID’s and global ID’s? For 
example, if PETSc is configured for 32-bit indices on very large problems with 
say 10 billion degrees of freedoms (exceeding the limit of 32-bit indices for a 
global ID) would it crash and require us to re-compile with 64-bit indices or 
can it gracefully handle this?

You would need to reconfigure for 64-bit indices. Keeping local indices 32-bit 
does not do much. I experimented with a local-index-only version several years 
ago, but it breaks a lot of things, and does not really deliver much benefit.

  Thanks,

Matt

Sincerely,
Brian


--
What most experimenters take for granted before they begin their experiments is 
infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/

Re: [petsc-users] Data types for local ID's and global ID's for large problems

[Matthew Knepley via petsc-users]

On Wed, Jan 9, 2019 at 5:53 PM Weston, Brian Thomas via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Does PETSc have different data types for local ID’s and global ID’s? For
> example, if PETSc is configured for 32-bit indices on very large problems
> with say 10 billion degrees of freedoms (exceeding the limit of 32-bit
> indices for a global ID) would it crash and require us to re-compile with
> 64-bit indices or can it gracefully handle this?
>

You would need to reconfigure for 64-bit indices. Keeping local indices
32-bit does not do much. I experimented with a local-index-only version
several years ago, but it breaks a lot of things, and does not really
deliver much benefit.

  Thanks,

Matt


> Sincerely,
>
> Brian
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

[petsc-users] Data types for local ID's and global ID's for large problems

[Weston, Brian Thomas via petsc-users]

Does PETSc have different data types for local ID’s and global ID’s? For 
example, if PETSc is configured for 32-bit indices on very large problems with 
say 10 billion degrees of freedoms (exceeding the limit of 32-bit indices for a 
global ID) would it crash and require us to re-compile with 64-bit indices or 
can it gracefully handle this?

Sincerely,
Brian