Hi Miguel,
On Tue, 1 Oct 2024 at 09:05, MIGUEL MOLINOS PEREZ <mmoli...@us.es> wrote:
> Sorry, I forgot to add:
>
> You think VecScatterCreate is the way to go?
>
No.
The semantics you seek are very similar to what VecGhost provides.
I was only using VecGhostUpdateBegin/End by way of example.
Cheers,
Dave
> Thanks,
> Miguel
>
> On Oct 1, 2024, at 5:56 PM, MIGUEL MOLINOS PEREZ <mmoli...@us.es> wrote:
>
> Hi Dave,
>
> Would something like that work?
>
> Yes, this should work! Any idea on where to look so I can try to implement
> it myself?
>
> Best,
> Miguel
>
> On Oct 1, 2024, at 5:22 PM, Dave May <dave.mayhe...@gmail.com> wrote:
>
> Hi Miguel,
>
> On Tue 1. Oct 2024 at 07:56, MIGUEL MOLINOS PEREZ <mmoli...@us.es> wrote:
>
>> Thank you Matt, it works!
>>
>> The implementation is straightforward:
>> - 1º Define the paddle regions using DMGetLocalBoundingBox with the
>> background DMDA mesh as an auxiliary mesh for the domain-partitioning.
>> - 2º Create an integer to count the local number of particles to be used
>> as ghost particle for other processors (N_ghost). One particle can be
>> counted more than one time. At the same time, fill two arrays:
>> - one with the index of the "main particle” (local particle),
>> - and the other with the target rank of the "main particle”.
>> - 3º Create the new particles using DMSwarmAddNPoints
>> - 4º Fill the new particles with the information of the “main particle”
>> but set the internal variable DMSwarmField_rank with the target rank.
>> - 5º Call DMSwarmMigrate(*,PETSC_TRUE). Therefore, we send the ghost
>> particles to the corresponding processors and we delete them from the
>> original processor.
>> - 6º Do stuff…
>> - 7º Delete ghost particles. This is very easy, we just have to call
>> DMSwarmRemovePoint N_ghost times.
>>
>> I think this can be easily implemented as closed routine for the DMSwarm
>> class.
>>
>> The remaining question is: how to do the communication between the
>> “original" particle and the ghost particles? For instance, if we update
>> some particle variable (locally) inside of a SNES context, this same
>> variable should be updated in the ghost particles at the other processors.
>>
>
> I think what you are asking about is an operation similar to
> VecGhostUpdate{Begin,End}(). In the case of a DMSwarm I’m not sure how to
> define the InsertMode = ADD_VALUES? Some swarm fields do not make sense to
> be added. INSERT_VALUES is fine.
>
> One solution might be to have something like this
>
> DMSwarmCollectViewUpdateGhostOwners(DM dm, InsertMode mode, PetscInt
> nfields, const char *fieldNames[]);
>
> where one can specify the insert mode and the fields on which the insert
> mode will apply.
>
> Would something like that work?
>
> Cheers,
> Dave
>
>
>
>
>
>> PS: Hope this helps someone in the future :-)
>>
>>
>> On Sep 27, 2024, at 10:50 AM, MIGUEL MOLINOS PEREZ <mmoli...@us.es>
>> wrote:
>>
>> Thank you Matt, let me give it try.
>>
>> Miguel
>>
>> On Sep 27, 2024, at 3:44 AM, Matthew Knepley <knep...@gmail.com> wrote:
>>
>> On Thu, Sep 26, 2024 at 7:18 PM MIGUEL MOLINOS PEREZ <mmoli...@us.es>
>> wrote:
>>
>>> I see, you mean:
>>>
>>> Create the ghost particles at the local cell with the same properties as
>>> particle 1 (duplicate the original particle) but different value
>>> DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do
>>> the migration and delete the local copies of the particle 1. Right?
>>>
>>
>> Yep. I think it will work, from what I know about BASIC.
>>
>> Thanks,
>>
>> Matt
>>
>>
>>> Thanks,
>>> Miguel
>>>
>>> On Sep 26, 2024, at 11:09 PM, Matthew Knepley <knep...@gmail.com> wrote:
>>>
>>> On Thu, Sep 26, 2024 at 11:20 AM MIGUEL MOLINOS PEREZ <mmoli...@us.es>
>>> wrote:
>>>
>>>> Thank you Matt.
>>>>
>>>> Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic
>>>> implementation
>>>> to see if there is some workaround.
>>>>
>>>> The idea of adding new particles is interesting. However, in that case,
>>>> we need to initialize the new (ghost) particles using the fields of
>>>> the “real” particle, right? This can be done using something like:
>>>>
>>>> VecGhostUpdateBegin
>>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhostUpdateBegin/__;!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk63job1UA$
>>>> >(Vec
>>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk6wexE7L8$
>>>> > globalout,InsertMode
>>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertMode/__;!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk63Qjcg-m$
>>>> > ADD_VALUES
>>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VALUES/__;!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk6364wZBs$
>>>> >, ScatterMode
>>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/ScatterMode/__;!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk670gaETR$
>>>> > SCATTER_REVERSE
>>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER_REVERSE/__;!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk69jjn4Yl$
>>>> >);VecGhostUpdateEnd
>>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhostUpdateEnd/__;!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk63EFcTKh$
>>>> >(Vec
>>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk6wexE7L8$
>>>> > globalout,InsertMode
>>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertMode/__;!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk63Qjcg-m$
>>>> > ADD_VALUES
>>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VALUES/__;!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk6364wZBs$
>>>> >, ScatterMode
>>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/ScatterMode/__;!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk670gaETR$
>>>> > SCATTER_REVERSE
>>>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER_REVERSE/__;!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk69jjn4Yl$
>>>> >);
>>>>
>>>> for the particle fields (?).
>>>>
>>>
>>> I think we can just copy from the local particle. For example, suppose I
>>> decide that particle 1 should go to rank 5, 12, and 27. Then
>>> I first set p1.rank = 5, then I add two new particles with the same
>>> values as particle 1, but with rank = 12 and 27. Then when I call migrate,
>>> it will move these three particles to the correct processes, and delete the
>>> original particles and the copies from the local set.
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>>
>>>> Thanks,
>>>> Miguel
>>>>
>>>>
>>>> On Sep 26, 2024, at 3:53 PM, Matthew Knepley <knep...@gmail.com> wrote:
>>>>
>>>> On Thu, Sep 26, 2024 at 6:31 AM MIGUEL MOLINOS PEREZ <mmoli...@us.es>
>>>> wrote:
>>>>
>>>>> Hi Matt et al,
>>>>>
>>>>> I’ve been working on the scheme that you proposed to create ghost
>>>>> particles (atoms in my case), and it works! With a couple of caveats:
>>>>> -1º In general the overlap particles will be migrate from their own
>>>>> rank to more than one neighbor rank, this is specially relevant for those
>>>>> located close to the corners. Therefore, you'll need to call
>>>>> DMSwarmMigrate
>>>>> several times (27 times for 3D cells), during the migration process.
>>>>>
>>>>
>>>> That is terrible. Let's just fix DMSwarmMigrate to have a mode that
>>>> sends the particle to all overlapping neighbors at once. It can't be that
>>>> hard.
>>>>
>>>>
>>>>> -2º You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed
>>>>> algorithm will not work at all!
>>>>>
>>>>
>>>> Oh, I should have thought of that. Sorry.
>>>>
>>>> I can help code up that extension. Can you take a quick look at the
>>>> BASIC code? Right now, we just use the rank attached to the particle
>>>> to send it. We could have an arrays of ranks, but that seems crazy, and
>>>> would blow up particle storage. How about just adding new particles
>>>> with the other ranks right before migration?
>>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>>
>>>>> Hope this helps to other folks!
>>>>>
>>>>> I have a follow-up question about periodic bcc on this context, should
>>>>> I open a new thread of keep posting here?
>>>>>
>>>>> Thanks,
>>>>> Miguel
>>>>>
>>>>> On Aug 7, 2024, at 4:22 AM, MIGUEL MOLINOS PEREZ <mmoli...@us.es>
>>>>> wrote:
>>>>>
>>>>> Thanks Matt, I think I'll start by making a small program as a proof
>>>>> of concept. Then, if it works I'll implement it in my code and I'll be
>>>>> happy to share it too :-)
>>>>>
>>>>> Miguel
>>>>>
>>>>> On Aug 4, 2024, at 3:30 AM, Matthew Knepley <knep...@gmail.com> wrote:
>>>>>
>>>>> On Fri, Aug 2, 2024 at 7:15 PM MIGUEL MOLINOS PEREZ <mmoli...@us.es>
>>>>> wrote:
>>>>>
>>>>>> Thanks again Matt, that makes a lot more sense !!
>>>>>>
>>>>>> Just to check that we are on the same page. You are saying:
>>>>>>
>>>>>> 1. create a field define a field called "owner rank" for each
>>>>>> particle.
>>>>>>
>>>>>> 2. Identify the phantom particles and modify the internal variable
>>>>>> defined by the DMSwarmField_rank variable.
>>>>>>
>>>>>> 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the
>>>>>> new local vector including the ghost particles.
>>>>>>
>>>>>> 4. Then, once the calculations are done, rename the DMSwarmField_rank
>>>>>> variable using the "owner rank" variable and call
>>>>>> DMSwarmMigrate(*,PETSC_FALSE) once again.
>>>>>>
>>>>>
>>>>> I don't think we need this last step. We can just remove those ghost
>>>>> particles for the next step I think.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Matt
>>>>>
>>>>>
>>>>>> Thank you,
>>>>>> Miguel
>>>>>>
>>>>>>
>>>>>> On Aug 2, 2024, at 5:33 PM, Matthew Knepley <knep...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> On Fri, Aug 2, 2024 at 11:15 AM MIGUEL MOLINOS PEREZ <mmoli...@us.es>
>>>>>> wrote:
>>>>>>
>>>>>>> Thank you Matt for your time,
>>>>>>>
>>>>>>> What you describe seems to me the ideal approach.
>>>>>>>
>>>>>>> 1) Add a particle field 'ghost' that identifies ghost vs owned
>>>>>>> particles. I think it needs options OWNED, OVERLAP, and GHOST
>>>>>>>
>>>>>>> This means, locally, I need to allocate Nlocal + ghost particles
>>>>>>> (duplicated) for my model?
>>>>>>>
>>>>>>
>>>>>> I would do it another way. I would allocate the particles with no
>>>>>> overlap and set them up. Then I would identify the halo particles, mark
>>>>>> them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles
>>>>>> as
>>>>>> GHOST, then unmark the OVERLAP particles. Shoot! That marking will not
>>>>>> work
>>>>>> since we cannot tell the difference between particles we received and
>>>>>> particles we sent. Okay, instead of the `ghost` field we need an `owner
>>>>>> rank` field. So then we
>>>>>>
>>>>>> 1) Setup the non-overlapping particles
>>>>>>
>>>>>> 2) Identify the halo particles
>>>>>>
>>>>>> 3) Change the `rank`, but not the `owner rank`
>>>>>>
>>>>>> 4) Call DMSwarmMigrate()
>>>>>>
>>>>>> Now we can identify ghost particles by the `owner rank`
>>>>>>
>>>>>>
>>>>>>> If that so, how to do the communication between the ghost particles
>>>>>>> living in the rank i and their “real” counterpart in the rank j.
>>>>>>>
>>>>>>> Algo, as an alternative, what about:
>>>>>>> 1) Use an IS tag which contains, for each rank, a list of the
>>>>>>> global index of the neighbors particles outside of the rank.
>>>>>>> 2) Use VecCreateGhost to create a new vector which contains extra
>>>>>>> local space for the ghost components of the vector.
>>>>>>> 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do
>>>>>>> the transference of data between a vector obtained with
>>>>>>> DMSwarmCreateGlobalVectorFromField
>>>>>>> 4) Do necessary computations using the vectors created with
>>>>>>> VecCreateGhost.
>>>>>>>
>>>>>>
>>>>>> This is essentially what Migrate() does. I was trying to reuse the
>>>>>> code.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>> Miguel
>>>>>>>
>>>>>>> On Aug 2, 2024, at 8:58 AM, Matthew Knepley <knep...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> On Thu, Aug 1, 2024 at 4:40 PM MIGUEL MOLINOS PEREZ <mmoli...@us.es>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> This Message Is From an External Sender
>>>>>>>> This message came from outside your organization.
>>>>>>>>
>>>>>>>>
>>>>>>>> Dear all,
>>>>>>>>
>>>>>>>> I am implementing a Molecular Dynamics (MD) code using the DMSWARM
>>>>>>>> interface. In the MD simulations we evaluate on each particle (atoms)
>>>>>>>> some kind of scalar functional using data from the neighbouring atoms.
>>>>>>>> My problem lies in the parallel implementation of the model, because
>>>>>>>> sometimes, some of these neighbours lie on a different processor.
>>>>>>>>
>>>>>>>> This is usually solved by using ghost particles. A similar approach
>>>>>>>> (with nodes instead) is already implemented for other PETSc mesh
>>>>>>>> structures like DMPlexConstructGhostCells. Unfortunately, I don't see
>>>>>>>> this kind of constructs for DMSWARM. Am I missing something?
>>>>>>>>
>>>>>>>> I this could be done by applying a buffer region by exploiting the
>>>>>>>> background DMDA mesh that I already use to do domain decomposition.
>>>>>>>> Then using the buffer region of each cell to locate the ghost
>>>>>>>> particles and finally using VecCreateGhost. Is this feasible? Or is
>>>>>>>> there an easier approach using other PETSc functions.
>>>>>>>>
>>>>>>>>
>>>>>>> This is feasible, but it would be good to develop a set of best
>>>>>>> practices, since we have been mainly focused on the case of
>>>>>>> non-redundant
>>>>>>> particles. Here is how I think I would do what you want.
>>>>>>>
>>>>>>> 1) Add a particle field 'ghost' that identifies ghost vs owned
>>>>>>> particles. I think it needs options OWNED, OVERLAP, and GHOST
>>>>>>>
>>>>>>> 2) At some interval identify particles that should be sent to other
>>>>>>> processes as ghosts. I would call these "overlap particles". The
>>>>>>> determination
>>>>>>> seems application specific, so I would leave this determination
>>>>>>> to the user right now. We do two things to these particles
>>>>>>>
>>>>>>> a) Mark chosen particles as OVERLAP
>>>>>>>
>>>>>>> b) Change rank to process we are sending to
>>>>>>>
>>>>>>> 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion
>>>>>>> flag
>>>>>>>
>>>>>>> 4) Mark OVERLAP particles as GHOST when they arrive
>>>>>>>
>>>>>>> There is one problem in the above algorithm. It does not allow
>>>>>>> sending particles to multiple ranks. We would have to do this
>>>>>>> in phases right now, or make a small adjustment to the interface
>>>>>>> allowing replication of particles when a set of ranks is specified.
>>>>>>>
>>>>>>> THanks,
>>>>>>>
>>>>>>> Matt
>>>>>>>
>>>>>>>
>>>>>>>> Thank you,
>>>>>>>> Miguel
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which
>>>>>>> their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk63NPQ60c$
>>>>>>>
>>>>>>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk62oSLJR_$
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which
>>>>>> their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk63NPQ60c$
>>>>>>
>>>>>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk62oSLJR_$
>>>>>> >
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk63NPQ60c$
>>>>>
>>>>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk62oSLJR_$
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk63NPQ60c$
>>>>
>>>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk62oSLJR_$
>>>> >
>>>>
>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk63NPQ60c$
>>>
>>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk62oSLJR_$
>>> >
>>>
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk63NPQ60c$
>>
>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aIhw4GigszsZG9HmvzDp8Lxt-OKLh4BxuMAUcfn8901TVodKwWeu7X3DFBjBZ5X0IGbICPb6T0PIj0GDVOAk62oSLJR_$
>> >
>>
>>
>
>
