[petsc-dev] What's the point of D(A/M)GetGlobalVector?
On Sun, 29 Aug 2010 09:16:18 -0400, Kai Germaschewski kai.germaschewski at
unh.edu wrote:
Some malloc()'s, like GNU libc, allow you to hook into it. Even if not, if
you define your own malloc(), it'll get used instead of the C library one
(depends on the order of libs on the command line). However, in that case
you may have trouble to do your thing and then continue on to the original
version. Actually, with dynamic libs you can do it:
http://stackoverflow.com/questions/262439/create-a-wrapper-function-for-malloc-and-free-in-c
I don't know how to handle C++, I imagine there's a way to do it on common
systems.
GNU new calls malloc, though not necessarily every time. Unfortunately
this is all non-portable, changes the linking procedure, and is not
appropriate for a library to do.
Note also that object creation and destruction are collective
operations, but malloc() may be called from user code on one process.
This may require GC'ing an unused cache, which involves destroying
parallel objects.
I take your point, it'll be major hassle, and it's quite likely you can't
make it work on every system. However, you could check at configure time,
and just not use the cache but free() things on destroy() if the system
doesn't support it. I'm also not sure I like the random timing distortion
caused by having something GC-like.
Before considering the complex solutions, I think it's important to first
answer
(1) Is the cache something that really make a difference in performance? (I
suppose whenever someone DAGet{Global,Local}Vector(), they will do something
to fill that vector with values, if that near always takes than even a
VecCreate(), there may be no point in caching those things in the first
place.
Depends on the size of the vector, coarse level residual evaluation
works with very small vectors, you'd rather not add a few collective
operations to that operation.
(2) Is there an easier way to fix the problem? One way may be to look at why
a DACreate{Local,Global}Vector() isn't performing as well as we'd need, and
there may ways to streamline/optimize it for the common case. A second way
could be a do_cache flag on Vec's (and maybe Mat's, too). That is, do your
VecCreate(), or DACreate{...}Vector(), and then VecSetCached(vec). That flag
would be checked at Destroy() time and the Vector would go into the pool
instead of being free'd. In that case, you don't eat up all mem for the
unsuspecting user, but an advanced user would easily be able to use the
benefit of caching where it actually matters (That's not as nice as having
everything fully automatic, but...)
If this cache is specific to the DA, then I don't see this being
preferable to DAGet*. If the cache is not specific to the DA, then you
need to somehow match compatible Vecs lying around in the cache, and I
still don't think the interface is as nice, and it seems less precise
when you want to reclaim that memory.
As an aside, talloc (http://talloc.samba.org) is an interesting
hierarchical, reference counting allocator. You allocate hierarchically
(new memory references a parent), and then automatically collect all
children when the parent is freed. There is support for various sharing
schemes, the interesting thing is that it makes it much easier to clean
up gracefully in error conditions. If you're interesting in C memory
management systems, it's worth a look.
Jed
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
I would support a name change to Create(). However, we should be really sure before we do stuff like that since nothing pisses people off (Wolfgang) like name changes. Matt On Fri, Aug 27, 2010 at 5:49 PM, Barry Smith bsmith at mcs.anl.gov wrote: Hmmm, petscda.h:EXTERN PetscErrorCode PETSCDM_DLLEXPORT DMGetColoring(DM,ISColoringType,const MatType,ISColoring*); petscda.h:EXTERN PetscErrorCode PETSCDM_DLLEXPORT DMGetMatrix(DM, const MatType,Mat*); petscda.h:EXTERN PetscErrorCode PETSCDM_DLLEXPORT DMGetInterpolation(DM,DM,Mat*,Vec*); petscda.h:EXTERN PetscErrorCode PETSCDM_DLLEXPORT DMGetInterpolationScale(DM,DM,Mat,Vec*); petscda.h:EXTERN PetscErrorCode PETSCDM_DLLEXPORT DMGetAggregates(DM,DM,Mat*); should all of these be Create? In my mind usually Get means get something intrinsic to the underlying object (some property of it for example); Create means generate a new thing that while it may be associated with the DA is not owned or controlled by the DA. Another way to organize is Create() implies you later Destroy() that object, while for things you Get you do something else (like restore). I'm inclined to change all of these ones to Create() since they are all Destroyed() Barry On Aug 27, 2010, at 11:10 AM, Jed Brown wrote: On Fri, 27 Aug 2010 12:00:32 -0400, Kai Germaschewski kai.germaschewski at unh.edu wrote: And it also requires some more memory management framework which would call upon caches to expire long-unused objects when memory is running low. How would you detect this? Note that further allocation may be done external to PETSc, and perhaps even in a separate process. We're not in a managed environment, we can't get a reliable time to GC. If we could get that sort of signal, then I would be for such caching at all times, but I don't think we can, in which case I still think managed/pooled access versus owned creation needs to be explicitly different. Jed -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- next part -- An HTML attachment was scrubbed... URL: http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20100828/0ed5ce22/attachment.html
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
I'm posting this to -dev since I'm working in -dev, let me know if I should be pushing this sort of question to -users. I was debugging a code today and noticed the following note in DAGetGlobalVector: The vector values are NOT initialized and may have garbage in them, so you may need to zero them. What exactly is the purpose of these routines then? Is there a global Vector associated with a DA? If so, why are the values uninitialized? On the other hand, if there isn't one, what's the sense of 'get/restore'? The following code does NOT work the way I'd expect: DAGetGlobalVector(x) /* modify x */ DARestoreGlobalVector(x) DAGetGlobalVector(x) /* access previous values, except everything is zero*/ I guess I'm not grokking something about the concept of a DA or PETSc objects, could somebody explain the purpose or correct usage of Get/Restore here. Thanks in advance, Aron -- next part -- An HTML attachment was scrubbed... URL: http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20100827/268d23b4/attachment.html
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
On Fri, 27 Aug 2010 14:13:01 +0300, Aron Ahmadia aron.ahmadia at kaust.edu.sa wrote: What exactly is the purpose of these routines then? Is there a global Vector associated with a DA? If so, why are the values uninitialized? It's common to need work vectors in places like residual evaluation and Jacobian assembly. There is a little bit of setup cost to allocate a new vector each time, so usually we'd prefer that they be persistent and just reuse them. One option would be to make the user manage this themselves, but that's error prone because it's easy to accidentally alias the work vectors, so instead the DA keeps a cache of vectors. It starts out empty, and each time you call DAGetGlobalVector(), the cache is searched for an available vector. If none are found, a new one is allocated and the cache grows by one. DARestoreGlobalVector() checks a vector back in so it may be used elsewhere. These vectors are destroyed in DADestroy(). Jed
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
Simply, in PETSc, getFoo() and restoreFoo() operate an object pool. Matt On Fri, Aug 27, 2010 at 11:23 AM, Jed Brown jed at 59a2.org wrote: On Fri, 27 Aug 2010 14:13:01 +0300, Aron Ahmadia aron.ahmadia at kaust.edu.sa wrote: What exactly is the purpose of these routines then? Is there a global Vector associated with a DA? If so, why are the values uninitialized? It's common to need work vectors in places like residual evaluation and Jacobian assembly. There is a little bit of setup cost to allocate a new vector each time, so usually we'd prefer that they be persistent and just reuse them. One option would be to make the user manage this themselves, but that's error prone because it's easy to accidentally alias the work vectors, so instead the DA keeps a cache of vectors. It starts out empty, and each time you call DAGetGlobalVector(), the cache is searched for an available vector. If none are found, a new one is allocated and the cache grows by one. DARestoreGlobalVector() checks a vector back in so it may be used elsewhere. These vectors are destroyed in DADestroy(). Jed -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- next part -- An HTML attachment was scrubbed... URL: http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20100827/3e5116e5/attachment.html
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
Thanks Matt and Jed, I think I'm straight on usage/philosophy here. A On Fri, Aug 27, 2010 at 2:42 PM, Matthew Knepley knepley at gmail.com wrote: Simply, in PETSc, getFoo() and restoreFoo() operate an object pool. Matt On Fri, Aug 27, 2010 at 11:23 AM, Jed Brown jed at 59a2.org wrote: On Fri, 27 Aug 2010 14:13:01 +0300, Aron Ahmadia aron.ahmadia at kaust.edu.sa wrote: What exactly is the purpose of these routines then? Is there a global Vector associated with a DA? If so, why are the values uninitialized? It's common to need work vectors in places like residual evaluation and Jacobian assembly. There is a little bit of setup cost to allocate a new vector each time, so usually we'd prefer that they be persistent and just reuse them. One option would be to make the user manage this themselves, but that's error prone because it's easy to accidentally alias the work vectors, so instead the DA keeps a cache of vectors. It starts out empty, and each time you call DAGetGlobalVector(), the cache is searched for an available vector. If none are found, a new one is allocated and the cache grows by one. DARestoreGlobalVector() checks a vector back in so it may be used elsewhere. These vectors are destroyed in DADestroy(). Jed -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- next part -- An HTML attachment was scrubbed... URL: http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20100827/dcd73c11/attachment.html
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
Except VecGetArray, etc, which operate a pool of one object. I think this may be the root cause of confusion. Dmitry. On Fri, Aug 27, 2010 at 6:42 AM, Matthew Knepley knepley at gmail.com wrote: Simply, in PETSc, getFoo() and restoreFoo() operate an object pool. ?? Matt On Fri, Aug 27, 2010 at 11:23 AM, Jed Brown jed at 59a2.org wrote: On Fri, 27 Aug 2010 14:13:01 +0300, Aron Ahmadia aron.ahmadia at kaust.edu.sa wrote: What exactly is the purpose of these routines then? ?Is there a global Vector associated with a DA? ?If so, why are the values uninitialized? It's common to need work vectors in places like residual evaluation and Jacobian assembly. ?There is a little bit of setup cost to allocate a new vector each time, so usually we'd prefer that they be persistent and just reuse them. ?One option would be to make the user manage this themselves, but that's error prone because it's easy to accidentally alias the work vectors, so instead the DA keeps a cache of vectors. ?It starts out empty, and each time you call DAGetGlobalVector(), the cache is searched for an available vector. ?If none are found, a new one is allocated and the cache grows by one. ?DARestoreGlobalVector() checks a vector back in so it may be used elsewhere. ?These vectors are destroyed in DADestroy(). Jed -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
Not to mention the various Get routines that are actually used to create things, such as DAGetMatrix. Still, the idea of a pool of work vectors makes sense, I was just trying to wrap my head around the actual use for those routines. A On Fri, Aug 27, 2010 at 4:06 PM, Dmitry Karpeev karpeev at mcs.anl.gov wrote: Except VecGetArray, etc, which operate a pool of one object. I think this may be the root cause of confusion. Dmitry. On Fri, Aug 27, 2010 at 6:42 AM, Matthew Knepley knepley at gmail.com wrote: Simply, in PETSc, getFoo() and restoreFoo() operate an object pool. Matt On Fri, Aug 27, 2010 at 11:23 AM, Jed Brown jed at 59a2.org wrote: On Fri, 27 Aug 2010 14:13:01 +0300, Aron Ahmadia aron.ahmadia at kaust.edu.sa wrote: What exactly is the purpose of these routines then? Is there a global Vector associated with a DA? If so, why are the values uninitialized? It's common to need work vectors in places like residual evaluation and Jacobian assembly. There is a little bit of setup cost to allocate a new vector each time, so usually we'd prefer that they be persistent and just reuse them. One option would be to make the user manage this themselves, but that's error prone because it's easy to accidentally alias the work vectors, so instead the DA keeps a cache of vectors. It starts out empty, and each time you call DAGetGlobalVector(), the cache is searched for an available vector. If none are found, a new one is allocated and the cache grows by one. DARestoreGlobalVector() checks a vector back in so it may be used elsewhere. These vectors are destroyed in DADestroy(). Jed -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- next part -- An HTML attachment was scrubbed... URL: http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20100827/39eceb50/attachment.html
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
On Fri, 27 Aug 2010 16:13:14 +0300, Aron Ahmadia aron.ahmadia at kaust.edu.sa
wrote:
Not to mention the various Get routines that are actually used to create
things, such as DAGetMatrix.
I think that should have been named DACreateMatrix(). Other XGetY() are
just accessors which create a managed object if needed. When there is a
Get/Restore pair, it implies that the access to that managed resource
has some sort of exclusivity. I think
{Get,Restore}{Local,Global}Vector() are actually the only functions with
that naming scheme that really manage a pool.
Jed
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
I agree with DACreateMatrix().
Matt
On Fri, Aug 27, 2010 at 1:24 PM, Jed Brown jed at 59a2.org wrote:
On Fri, 27 Aug 2010 16:13:14 +0300, Aron Ahmadia
aron.ahmadia at kaust.edu.sa wrote:
Not to mention the various Get routines that are actually used to create
things, such as DAGetMatrix.
I think that should have been named DACreateMatrix(). Other XGetY() are
just accessors which create a managed object if needed. When there is a
Get/Restore pair, it implies that the access to that managed resource
has some sort of exclusivity. I think
{Get,Restore}{Local,Global}Vector() are actually the only functions with
that naming scheme that really manage a pool.
Jed
--
What most experimenters take for granted before they begin their experiments
is infinitely more interesting than any results to which their experiments
lead.
-- Norbert Wiener
-- next part --
An HTML attachment was scrubbed...
URL:
http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20100827/3c4594d7/attachment.html
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
On Aug 27, 2010, at 8:13 AM, Aron Ahmadia wrote: Not to mention the various Get routines that are actually used to create things, such as DAGetMatrix. Still, the idea of a pool of work vectors makes sense, I was just trying to wrap my head around the actual use for those routines. Aron, DAGetMatrix() is actually a bug and should be DACreateMatrix() (or maybe better DACreateMat() while the DACreateGlobalVector() and friends should really be DACreateGlobalVec()). Are there others beside DAGetMatrix() that are incorrect with gets that should be creates? Thanks Barry It would actually be nice if we made DACreateGlobal/LocalVector() so light-weight that it could be used for work vectors (instead of needing a different set of light weight get routines) but then we would need DADestroyGlobal/Vector() to handle putting back in the free list or need to modify VecDestroy() to handle not actually destroying but managing a free list). And there is also the issue of zeroing or not zeroing the Vec initially. This is why we still have the Create and Get versions. A On Fri, Aug 27, 2010 at 4:06 PM, Dmitry Karpeev karpeev at mcs.anl.gov wrote: Except VecGetArray, etc, which operate a pool of one object. I think this may be the root cause of confusion. Dmitry. On Fri, Aug 27, 2010 at 6:42 AM, Matthew Knepley knepley at gmail.com wrote: Simply, in PETSc, getFoo() and restoreFoo() operate an object pool. Matt On Fri, Aug 27, 2010 at 11:23 AM, Jed Brown jed at 59a2.org wrote: On Fri, 27 Aug 2010 14:13:01 +0300, Aron Ahmadia aron.ahmadia at kaust.edu.sa wrote: What exactly is the purpose of these routines then? Is there a global Vector associated with a DA? If so, why are the values uninitialized? It's common to need work vectors in places like residual evaluation and Jacobian assembly. There is a little bit of setup cost to allocate a new vector each time, so usually we'd prefer that they be persistent and just reuse them. One option would be to make the user manage this themselves, but that's error prone because it's easy to accidentally alias the work vectors, so instead the DA keeps a cache of vectors. It starts out empty, and each time you call DAGetGlobalVector(), the cache is searched for an available vector. If none are found, a new one is allocated and the cache grows by one. DARestoreGlobalVector() checks a vector back in so it may be used elsewhere. These vectors are destroyed in DADestroy(). Jed -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- next part -- An HTML attachment was scrubbed... URL: http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20100827/27177aa0/attachment.html
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
On 27 August 2010 10:27, Barry Smith bsmith at mcs.anl.gov wrote: On Aug 27, 2010, at 8:13 AM, Aron Ahmadia wrote: Not to mention the various Get routines that are actually used to create things, such as?DAGetMatrix. ?Still, the idea of a pool of work vectors makes sense, I was just trying to wrap my head around the actual use for those routines. ?? Aron, ?? ?DAGetMatrix() is actually a bug and should be DACreateMatrix() ? (or maybe better DACreateMat() while the DACreateGlobalVector() and friends should really be DACreateGlobalVec()). I think you are right. ?? ?Are there others beside DAGetMatrix() that are incorrect with gets that should be creates? MatGetVecs() -- Lisandro Dalcin --- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
Barry, I was being snippy, DAGetMatrix is the only one I know of that acts unexpectedly. Still, when working with Vec, you expect Get/Restore to modify data in the Vec itself, which breaks for DAs since a DA has no internal Vec storage. Unless you are willing to append 'Work' somewhere into the names of the work vector routines, I don't see an obvious solution. A On Fri, Aug 27, 2010 at 4:27 PM, Barry Smith bsmith at mcs.anl.gov wrote: On Aug 27, 2010, at 8:13 AM, Aron Ahmadia wrote: Not to mention the various Get routines that are actually used to create things, such as DAGetMatrix. Still, the idea of a pool of work vectors makes sense, I was just trying to wrap my head around the actual use for those routines. Aron, DAGetMatrix() is actually a bug and should be DACreateMatrix() (or maybe better DACreateMat() while the DACreateGlobalVector() and friends should really be DACreateGlobalVec()). Are there others beside DAGetMatrix() that are incorrect with gets that should be creates? Thanks Barry It would actually be nice if we made DACreateGlobal/LocalVector() so light-weight that it could be used for work vectors (instead of needing a different set of light weight get routines) but then we would need DADestroyGlobal/Vector() to handle putting back in the free list or need to modify VecDestroy() to handle not actually destroying but managing a free list). And there is also the issue of zeroing or not zeroing the Vec initially. This is why we still have the Create and Get versions. A On Fri, Aug 27, 2010 at 4:06 PM, Dmitry Karpeev karpeev at mcs.anl.govwrote: Except VecGetArray, etc, which operate a pool of one object. I think this may be the root cause of confusion. Dmitry. On Fri, Aug 27, 2010 at 6:42 AM, Matthew Knepley knepley at gmail.com wrote: Simply, in PETSc, getFoo() and restoreFoo() operate an object pool. Matt On Fri, Aug 27, 2010 at 11:23 AM, Jed Brown jed at 59a2.org wrote: On Fri, 27 Aug 2010 14:13:01 +0300, Aron Ahmadia aron.ahmadia at kaust.edu.sa wrote: What exactly is the purpose of these routines then? Is there a global Vector associated with a DA? If so, why are the values uninitialized? It's common to need work vectors in places like residual evaluation and Jacobian assembly. There is a little bit of setup cost to allocate a new vector each time, so usually we'd prefer that they be persistent and just reuse them. One option would be to make the user manage this themselves, but that's error prone because it's easy to accidentally alias the work vectors, so instead the DA keeps a cache of vectors. It starts out empty, and each time you call DAGetGlobalVector(), the cache is searched for an available vector. If none are found, a new one is allocated and the cache grows by one. DARestoreGlobalVector() checks a vector back in so it may be used elsewhere. These vectors are destroyed in DADestroy(). Jed -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- next part -- An HTML attachment was scrubbed... URL: http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20100827/69ed0811/attachment.html
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
On Fri, 27 Aug 2010 08:27:22 -0500, Barry Smith bsmith at mcs.anl.gov wrote: It would actually be nice if we made DACreateGlobal/LocalVector() so light-weight that it could be used for work vectors (instead of needing a different set of light weight get routines) but then we would need DADestroyGlobal/Vector() to handle putting back in the free list or need to modify VecDestroy() to handle not actually destroying but managing a free list). Hmm, I'm not sure that only having Create versions would be a good thing. Overhead could be reduced to a VecDuplicate, but you still have to allocate the memory. Malloc is plenty fast (~400 cycles everywhere I've checked) if you don't touch the memory, but traversing it extra times is not ideal, yet the reproducibility of always zeroing newly created vectors is handy. Maybe there are places that malloc() is more expensive, or that fragmentation (if you have some very small vectors, or are using huge pages) would become a factor. If the vectors are going to be persistent, their life has to be managed somehow, in which case the Get versions are needed. It also helps (me) to clarify intent: if I see a Get, then I know the vector is temporary without needing to check for a matching Destroy. Jed
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
On 27 August 2010 10:47, Jed Brown jed at 59a2.org wrote: ?It also helps (me) to clarify intent: if I see a Get, then I know the vector is temporary without needing to check for a matching Destroy. You need a matching Restore() -- Lisandro Dalcin --- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
On Fri, 27 Aug 2010 10:56:56 -0300, Lisandro Dalcin dalcinl at gmail.com wrote: On 27 August 2010 10:47, Jed Brown jed at 59a2.org wrote: ?It also helps (me) to clarify intent: if I see a Get, then I know the vector is temporary without needing to check for a matching Destroy. You need a matching Restore() Right, but I know when I see the Get that there will be a matching Restore (I don't have to infer from context or looking ahead that the object isn't handed off in some other way). Jed
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
I think DAGetXXX etc should really be thought of as constructors,
that under the hood manage a pool to amortize the construction time.
Perhaps more precisely, DA acts as a factory.
It would be natural to rename them DACreateXXX, except that
then XXXDestroy is the wrong thing to do (unless the destructor
is overloaded in the corresponding object).
Dmitry.
On Fri, Aug 27, 2010 at 8:24 AM, Jed Brown jed at 59a2.org wrote:
On Fri, 27 Aug 2010 16:13:14 +0300, Aron Ahmadia aron.ahmadia at
kaust.edu.sa wrote:
Not to mention the various Get routines that are actually used to create
things, such as DAGetMatrix.
I think that should have been named DACreateMatrix(). ?Other XGetY() are
just accessors which create a managed object if needed. ?When there is a
Get/Restore pair, it implies that the access to that managed resource
has some sort of exclusivity. ?I think
{Get,Restore}{Local,Global}Vector() are actually the only functions with
that naming scheme that really manage a pool.
Jed
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
On Fri, 27 Aug 2010 09:45:36 -0500, Dmitry Karpeev karpeev at mcs.anl.gov wrote: I think DAGetXXX etc should really be thought of as constructors, that under the hood manage a pool to amortize the construction time. Perhaps more precisely, DA acts as a factory. It would be natural to rename them DACreateXXX, except that then XXXDestroy is the wrong thing to do (unless the destructor is overloaded in the corresponding object). Hmm, I think there is an important distinction, in terms of overall memory use, between objects that live a long time and those that do not. I might allocate a bunch of memory in a preprocessing stage, release it, and then build solver objects. If all that preprocessing memory stayed alive for the life of the program, I would run out of memory. You can't overload Destroy in place of Restore unless you maintain upward links or guarantee that the managing object will never need to do anything when you restore. One way to do this would be to double-reference gotten objects and consider them to be checked in any time the reference count drops to 1. But this excludes more elaborate data structures and extra consistency checks. I don't see any great nastiness of Get/Restore for managed objects and Create/Destroy for objects that the user wants to own. Jed
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
Maybe Create/Destroy isn't the right solution, but there appears to be some confusion about the meaning of Get/Restore in this context. It definitely differs from VecGet/RestoreArray. For example: there is no guarantee that subsequent DAGetXXXs, punctuated by DARestoreXXXs, will give one the same object. Dmitry. On Fri, Aug 27, 2010 at 10:02 AM, Jed Brown jed at 59a2.org wrote: On Fri, 27 Aug 2010 09:45:36 -0500, Dmitry Karpeev karpeev at mcs.anl.gov wrote: I think DAGetXXX etc should really be thought of as constructors, that under the hood manage a pool to amortize the construction time. Perhaps more precisely, DA acts as a factory. It would be natural to rename them DACreateXXX, except that then XXXDestroy is the wrong thing to do (unless the destructor is overloaded in the corresponding object). Hmm, I think there is an important distinction, in terms of overall memory use, between objects that live a long time and those that do not. I might allocate a bunch of memory in a preprocessing stage, release it, and then build solver objects. ?If all that preprocessing memory stayed alive for the life of the program, I would run out of memory. You can't overload Destroy in place of Restore unless you maintain upward links or guarantee that the managing object will never need to do anything when you restore. ?One way to do this would be to double-reference gotten objects and consider them to be checked in any time the reference count drops to 1. ?But this excludes more elaborate data structures and extra consistency checks. I don't see any great nastiness of Get/Restore for managed objects and Create/Destroy for objects that the user wants to own. Jed
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
On Fri, 27 Aug 2010 10:14:30 -0500, Dmitry Karpeev karpeev at mcs.anl.gov wrote: Maybe Create/Destroy isn't the right solution, but there appears to be some confusion about the meaning of Get/Restore in this context. It definitely differs from VecGet/RestoreArray. For example: there is no guarantee that subsequent DAGetXXXs, punctuated by DARestoreXXXs, will give one the same object. Indeed, managing a pool has different semantics. Is it worth looking for a less ambiguous/overloaded name? Checkout/Checkin Borrow/Return Claim/Release Jed
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
On Aug 27, 2010, at 8:47 AM, Jed Brown wrote: On Fri, 27 Aug 2010 08:27:22 -0500, Barry Smith bsmith at mcs.anl.gov wrote: It would actually be nice if we made DACreateGlobal/LocalVector() so light-weight that it could be used for work vectors (instead of needing a different set of light weight get routines) but then we would need DADestroyGlobal/Vector() to handle putting back in the free list or need to modify VecDestroy() to handle not actually destroying but managing a free list). Hmm, I'm not sure that only having Create versions would be a good thing. Overhead could be reduced to a VecDuplicate, but you still have to allocate the memory. Malloc is plenty fast (~400 cycles everywhere I've checked) if you don't touch the memory, but traversing it extra times is not ideal, yet the reproducibility of always zeroing newly created vectors is handy. Maybe there are places that malloc() is more expensive, or that fragmentation (if you have some very small vectors, or are using huge pages) would become a factor. If the DACreateVec managed a pool and the destroy put it back in a pool then you would not all this overhead. It is fine to have a DACreate and DAGet but I think it is possible to have just one. Barry If the vectors are going to be persistent, their life has to be managed somehow, in which case the Get versions are needed. It also helps (me) to clarify intent: if I see a Get, then I know the vector is temporary without needing to check for a matching Destroy. Jed
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
On Fri, 27 Aug 2010 10:50:22 -0500, Barry Smith bsmith at mcs.anl.gov wrote: If the DACreateVec managed a pool and the destroy put it back in a pool then you would not all this overhead. The problem with this is that it doesn't actually release the memory. So in the preprocessing scenario where I create, say, 50 local vectors, use them all temporarily, then destroy them and build a solver, I would run out of memory. I think there needs to be some way for the user to guarantee that the memory is actually released. Jed
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
I think it's conceptually not right to have Get()/Restore() pairs to do caching, but provide that feature only in special cases. Shouldn't other places that needs temp work vectors have a similar facility? I think a better way of handling this would be to get rid of the special Get()/Restore() caching pairs, and rather make caching a feature of the object itself. I.e., VecCreate() might have a pool of allocated vector data structures + associated data storage arrays. VecDestroy() would not really destroy a Vec but return it to the pool. Unfortunately that doesn't mix too well with how the real setup of a vector is done later, after options and potentially command line options have been processed, so by the time you know what kind of vector you want, you've pretty much already filled the data structure. And it also requires some more memory management framework which would call upon caches to expire long-unused objects when memory is running low. I think a more consistent user interface would be to just have DACreateVec(), and something like MatCreateVecs(), and then VecDestroy() the Vec when you're done, no matter where it came from. Whether it's cached or not is an implementation detail. Probably the first thing to figure out would be whether caching is making an actual difference in the real world, and if not, there's a pretty straight forward solution... --Kai On Fri, Aug 27, 2010 at 11:36 AM, Jed Brown jed at 59a2.org wrote: On Fri, 27 Aug 2010 10:14:30 -0500, Dmitry Karpeev karpeev at mcs.anl.gov wrote: Maybe Create/Destroy isn't the right solution, but there appears to be some confusion about the meaning of Get/Restore in this context. It definitely differs from VecGet/RestoreArray. For example: there is no guarantee that subsequent DAGetXXXs, punctuated by DARestoreXXXs, will give one the same object. Indeed, managing a pool has different semantics. Is it worth looking for a less ambiguous/overloaded name? Checkout/Checkin Borrow/Return Claim/Release Jed -- Kai Germaschewski Assistant Professor, Dept of Physics / Space Science Center University of New Hampshire, Durham, NH 03824 office: Morse Hall 245E phone: +1-603-862-2912 fax: +1-603-862-2771 -- next part -- An HTML attachment was scrubbed... URL: http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20100827/707ed688/attachment.html
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
On Fri, 27 Aug 2010 12:00:32 -0400, Kai Germaschewski kai.germaschewski at unh.edu wrote: And it also requires some more memory management framework which would call upon caches to expire long-unused objects when memory is running low. How would you detect this? Note that further allocation may be done external to PETSc, and perhaps even in a separate process. We're not in a managed environment, we can't get a reliable time to GC. If we could get that sort of signal, then I would be for such caching at all times, but I don't think we can, in which case I still think managed/pooled access versus owned creation needs to be explicitly different. Jed
[petsc-dev] What's the point of D(A/M)GetGlobalVector?
Hmmm, petscda.h:EXTERN PetscErrorCode PETSCDM_DLLEXPORT DMGetColoring(DM,ISColoringType,const MatType,ISColoring*); petscda.h:EXTERN PetscErrorCode PETSCDM_DLLEXPORT DMGetMatrix(DM, const MatType,Mat*); petscda.h:EXTERN PetscErrorCode PETSCDM_DLLEXPORT DMGetInterpolation(DM,DM,Mat*,Vec*); petscda.h:EXTERN PetscErrorCode PETSCDM_DLLEXPORT DMGetInterpolationScale(DM,DM,Mat,Vec*); petscda.h:EXTERN PetscErrorCode PETSCDM_DLLEXPORT DMGetAggregates(DM,DM,Mat*); should all of these be Create? In my mind usually Get means get something intrinsic to the underlying object (some property of it for example); Create means generate a new thing that while it may be associated with the DA is not owned or controlled by the DA. Another way to organize is Create() implies you later Destroy() that object, while for things you Get you do something else (like restore). I'm inclined to change all of these ones to Create() since they are all Destroyed() Barry On Aug 27, 2010, at 11:10 AM, Jed Brown wrote: On Fri, 27 Aug 2010 12:00:32 -0400, Kai Germaschewski kai.germaschewski at unh.edu wrote: And it also requires some more memory management framework which would call upon caches to expire long-unused objects when memory is running low. How would you detect this? Note that further allocation may be done external to PETSc, and perhaps even in a separate process. We're not in a managed environment, we can't get a reliable time to GC. If we could get that sort of signal, then I would be for such caching at all times, but I don't think we can, in which case I still think managed/pooled access versus owned creation needs to be explicitly different. Jed
