Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Jed Brown]

Ed Bueler  writes:

> satish> Perhaps if some of us get this (create branch) access at
> https://github.com/bueler/p4pdes - the workflow is slightly
> satish> simplified [and a fork can be avoided which would require toggle of
> giturl as the MR progresses between
> satish> fork url and upstream url - and the commit-ids that change as the
> upstream MR progresses..]
>
> I appreciate the desire to avoid working in forks, so I went ahead and
> invited Barry, Jed, Satish as collaborators on p4pdes.  Please let me know
> if there are others to invite.

Thanks. I don't think this materially changes the workflow, but it does
mean PETSc's configure doesn't need to learn how to switch remotes in
packages it downloads, and .../packages/p4pdes.py can just name commits so
long as review in bueler/p4pdes just results in new commits rather than
editing/rebasing the branch.

> Indeed, I suppose my preferred workflow on p4pdes is for collaborators to
> create branches and do MRs.  It looks like actually restricting
> collaborators from pushing on master is not something I can do.  ("The
> ability to restrict branches is a type of branch protection that's
> available for public and private repositories owned by organizations in
> GitHub Team, GitHub Enterprise Cloud, and GitHub Enterprise Server.")
> Other forms of protecting master look like they add steps to my own (direct
> and minimal) workflow on master.

You should be able to click "Add rule" here

https://github.com/bueler/p4pdes/settings/branches

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Ed Bueler]

satish> Perhaps if some of us get this (create branch) access at
https://github.com/bueler/p4pdes - the workflow is slightly
satish> simplified [and a fork can be avoided which would require toggle of
giturl as the MR progresses between
satish> fork url and upstream url - and the commit-ids that change as the
upstream MR progresses..]

I appreciate the desire to avoid working in forks, so I went ahead and
invited Barry, Jed, Satish as collaborators on p4pdes.  Please let me know
if there are others to invite.

Indeed, I suppose my preferred workflow on p4pdes is for collaborators to
create branches and do MRs.  It looks like actually restricting
collaborators from pushing on master is not something I can do.  ("The
ability to restrict branches is a type of branch protection that's
available for public and private repositories owned by organizations in
GitHub Team, GitHub Enterprise Cloud, and GitHub Enterprise Server.")
Other forms of protecting master look like they add steps to my own (direct
and minimal) workflow on master.

Ed


On Mon, Nov 2, 2020 at 6:32 AM Barry Smith  wrote:

>
>
> > On Nov 2, 2020, at 12:39 AM, Satish Balay  wrote:
> >
> > On Sun, 1 Nov 2020, Barry Smith wrote:
> >
> >>
> >>  Model: When working with PETSc master.
> >>
> >>The tests in a "master"  (not necessarily the exact master that the
> external person is using) of the external package run in the CI.
> >>
> >>Goal: The PETSc developer can debug and make changes to PETSc and to
> the external package to fix the programs in the most painless way possible.
> With the least number of steps, MR, and generally forgetting things.
> >>
> >>Example:  I change some PETSc code and the pipeline comes back with
> a failure on a p4pdes example.
> >>
> >>I debug the crashing p4pdes example and fix a problem in it.
> >>
> >>I now want to make a PETSc MR with my new PETSc code and update the
> copy of the external "master" so when my MR get's merged the pipelines will
> work with the external "master".
> >
> > We are still missing how we are syncing your fixes with p4pdes upstream.
> MR to upstream? (and likely it will have to through more revisions than
> what we had with slepc/petsc before getting merged.)
>
>We decide in advance if Ed wants it every fix, unlikely, or a monthly,
> or at release MR.  (This can be adjusted at any time). Yes we just make a
> MR to Ed's repository, he changes his repository based on the MR and we
> merge his new master into our fork. I don't see this as a big deal, no need
> for iterations since Ed decides how he wants his code to look.
> >
> > Note: if we are replicating the slepc [or prior petsc4py] workflow -
> then: the PETSc MR does not get merged until the upstream MR is merged. (so
> need this coordination) Is this acceptable with p4pdes?
>
>   No, I don't think we should do this with SLEPc either, it puts too much
> burden on Jose and slows things down. We should always point to our fork,
> never directly to p4pdes or SLEPc so everyone can work at their own pace
> asynchronously. Releases point to the true repository when possible, master
> points to the forks.
>
> >
> > And then there is the permissions issue. We had permissions to create
> branches in petsc4py repo - so could create these MRs easily (and have them
> compatible with --download-package). [don't remember if we have this access
> for slepc - but issues where slepc broke and need fixes on slepc side were
> rare - so we are able to wait for Jose's fix - before merging PETSc.]
>
>Since we will be doing this for other packages that won't give us
> permission I would always use a fork even when we don't have to. A fork of
> p4pdes, a fork of slepc,
> >
> > Perhaps if some of us get this (create branch) access at
> https://github.com/bueler/p4pdes - the workflow is slightly simplified
> [and a fork can be avoided which would require toggle of giturl as the MR
> progresses between fork url and upstream url - and the commit-ids that
> change as the upstream MR progresses..]
>
> If some people want to make the MR directly off the repository and
> have write permission they would just set the original origin to be the
> fork and when they want to make an MR then push to the remote that is the
> true repository.   I don't think we want or need a model where the "fork"
> is just the true repository, yes a tiny bit easier for some people some of
> the time.
>
>   All this is not as complex as you are making out. Basically it just does
> in a systematic way for all git packages what we do now in an ad hoc way.
>
>   Barry
>
> >
> >>
> >>  Other Goal:  The developer of the external package can update their
> code asynchronously with our work, that is if they delay we can keep on
> working on stuff on and on and we can update our test version
> asynchronously from their work.
> >
> > yes - this occurs with slepc [and occurred in older petsc4py model]
> >
> > Satish
> >
> >>
> >>  I proposed a plan with --download-xxx which

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Barry Smith]

> On Nov 2, 2020, at 12:39 AM, Satish Balay  wrote:
> 
> On Sun, 1 Nov 2020, Barry Smith wrote:
> 
>> 
>>  Model: When working with PETSc master.
>> 
>>The tests in a "master"  (not necessarily the exact master that the 
>> external person is using) of the external package run in the CI.
>> 
>>Goal: The PETSc developer can debug and make changes to PETSc and to the 
>> external package to fix the programs in the most painless way possible. With 
>> the least number of steps, MR, and generally forgetting things.
>> 
>>Example:  I change some PETSc code and the pipeline comes back with a 
>> failure on a p4pdes example.
>> 
>>I debug the crashing p4pdes example and fix a problem in it.
>> 
>>I now want to make a PETSc MR with my new PETSc code and update the copy 
>> of the external "master" so when my MR get's merged the pipelines will work 
>> with the external "master". 
> 
> We are still missing how we are syncing your fixes with p4pdes upstream. MR 
> to upstream? (and likely it will have to through more revisions than what we 
> had with slepc/petsc before getting merged.)

   We decide in advance if Ed wants it every fix, unlikely, or a monthly, or at 
release MR.  (This can be adjusted at any time). Yes we just make a MR to Ed's 
repository, he changes his repository based on the MR and we merge his new 
master into our fork. I don't see this as a big deal, no need for iterations 
since Ed decides how he wants his code to look.
> 
> Note: if we are replicating the slepc [or prior petsc4py] workflow - then: 
> the PETSc MR does not get merged until the upstream MR is merged. (so need 
> this coordination) Is this acceptable with p4pdes?

  No, I don't think we should do this with SLEPc either, it puts too much 
burden on Jose and slows things down. We should always point to our fork, never 
directly to p4pdes or SLEPc so everyone can work at their own pace 
asynchronously. Releases point to the true repository when possible, master 
points to the forks.

> 
> And then there is the permissions issue. We had permissions to create 
> branches in petsc4py repo - so could create these MRs easily (and have them 
> compatible with --download-package). [don't remember if we have this access 
> for slepc - but issues where slepc broke and need fixes on slepc side were 
> rare - so we are able to wait for Jose's fix - before merging PETSc.]

   Since we will be doing this for other packages that won't give us permission 
I would always use a fork even when we don't have to. A fork of p4pdes, a fork 
of slepc, 
> 
> Perhaps if some of us get this (create branch) access at 
> https://github.com/bueler/p4pdes - the workflow is slightly simplified [and a 
> fork can be avoided which would require toggle of giturl as the MR progresses 
> between fork url and upstream url - and the commit-ids that change as the 
> upstream MR progresses..]

If some people want to make the MR directly off the repository and have 
write permission they would just set the original origin to be the fork and 
when they want to make an MR then push to the remote that is the true 
repository.   I don't think we want or need a model where the "fork" is just 
the true repository, yes a tiny bit easier for some people some of the time.

  All this is not as complex as you are making out. Basically it just does in a 
systematic way for all git packages what we do now in an ad hoc way.

  Barry

> 
>> 
>>  Other Goal:  The developer of the external package can update their code 
>> asynchronously with our work, that is if they delay we can keep on working 
>> on stuff on and on and we can update our test version asynchronously from 
>> their work. 
> 
> yes - this occurs with slepc [and occurred in older petsc4py model]
> 
> Satish
> 
>> 
>>  I proposed a plan with --download-xxx which we already are comfortable with 
>> and understand. Jed proposed a plan with git tree we are not yet completely 
>> familiar with.
>> 
>>  Barry
>> 
>> 
>> 
>> 
>>> On Nov 1, 2020, at 8:56 PM, Satish Balay  wrote:
>>> 
>>> On Sun, 1 Nov 2020, Barry Smith wrote:
>>> 
 
 
> On Nov 1, 2020, at 2:57 PM, Jed Brown  wrote:
> 
> Barry Smith  writes:
> 
 I vote to fix things in our fork or gittree thing continuously since 
 it makes it easier to fix things rather than wait to the release when 
 we try to find and fix everything and it also helps tell us if we 
 introduced a real bug into PETSc and fix PETSc immediately instead of 
 waiting up to 6 months, just like we now we test immediately with 
 Petsc4py and we should do with SLEPc.  How often we give the updates 
 to Ed is a completely different issue. 
 
 So again back to my original statement ,it comes down to if the 
 subtree or the fork approach is easier for all the PETSc developers 
 who do not currently know gittree and would need to learn it with your 
 approach. I 

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Satish Balay via petsc-dev]

On Sun, 1 Nov 2020, Barry Smith wrote:

> 
>   Model: When working with PETSc master.
> 
> The tests in a "master"  (not necessarily the exact master that the 
> external person is using) of the external package run in the CI.
> 
> Goal: The PETSc developer can debug and make changes to PETSc and to the 
> external package to fix the programs in the most painless way possible. With 
> the least number of steps, MR, and generally forgetting things.
> 
> Example:  I change some PETSc code and the pipeline comes back with a 
> failure on a p4pdes example.
> 
> I debug the crashing p4pdes example and fix a problem in it.
> 
> I now want to make a PETSc MR with my new PETSc code and update the copy 
> of the external "master" so when my MR get's merged the pipelines will work 
> with the external "master". 

We are still missing how we are syncing your fixes with p4pdes upstream. MR to 
upstream? (and likely it will have to through more revisions than what we had 
with slepc/petsc before getting merged.)

Note: if we are replicating the slepc [or prior petsc4py] workflow - then: the 
PETSc MR does not get merged until the upstream MR is merged. (so need this 
coordination) Is this acceptable with p4pdes?

And then there is the permissions issue. We had permissions to create branches 
in petsc4py repo - so could create these MRs easily (and have them compatible 
with --download-package). [don't remember if we have this access for slepc - 
but issues where slepc broke and need fixes on slepc side were rare - so we are 
able to wait for Jose's fix - before merging PETSc.]

Perhaps if some of us get this (create branch) access at 
https://github.com/bueler/p4pdes - the workflow is slightly simplified [and a 
fork can be avoided which would require toggle of giturl as the MR progresses 
between fork url and upstream url - and the commit-ids that change as the 
upstream MR progresses..]

> 
>   Other Goal:  The developer of the external package can update their code 
> asynchronously with our work, that is if they delay we can keep on working on 
> stuff on and on and we can update our test version asynchronously from their 
> work. 

yes - this occurs with slepc [and occurred in older petsc4py model]

Satish

> 
>   I proposed a plan with --download-xxx which we already are comfortable with 
> and understand. Jed proposed a plan with git tree we are not yet completely 
> familiar with.
> 
>   Barry
> 
> 
> 
> 
> > On Nov 1, 2020, at 8:56 PM, Satish Balay  wrote:
> > 
> > On Sun, 1 Nov 2020, Barry Smith wrote:
> > 
> >> 
> >> 
> >>> On Nov 1, 2020, at 2:57 PM, Jed Brown  wrote:
> >>> 
> >>> Barry Smith  writes:
> >>> 
> >>  I vote to fix things in our fork or gittree thing continuously since 
> >> it makes it easier to fix things rather than wait to the release when 
> >> we try to find and fix everything and it also helps tell us if we 
> >> introduced a real bug into PETSc and fix PETSc immediately instead of 
> >> waiting up to 6 months, just like we now we test immediately with 
> >> Petsc4py and we should do with SLEPc.  How often we give the updates 
> >> to Ed is a completely different issue. 
> >> 
> >> So again back to my original statement ,it comes down to if the 
> >> subtree or the fork approach is easier for all the PETSc developers 
> >> who do not currently know gittree and would need to learn it with your 
> >> approach. I don't know which is easier learning to use gittree which 
> >> has its own gotcha's or using mine which we all know but may require 
> >> an extra step (not involving Ed, just updating the p4pdes.py commit 
> >> each time we change something in the fork.)
> >> 
> >> I think using --download-p4pdes on a couple of systems in the CI is 
> >> enough, I don't think we need to put it all CI pipelines (I would like 
> >> slepc in all pipelines).but we could put in all pipelines if we want.
> >> 
> >> For completeness I show the exact the work flow for my suggestion
> >> 
> >> pipelines --download-p4pdes and runs its tests
> >> if it breaks the developer uses --download-p4pdes  on their system 
> >>   they fix the problem either by fixing PETSc or what is downloaded 
> >> from the p4pde fork
> >>  if the fix is in the p4pdes fork they make a branch in the p4pdes 
> >> fork, which they already have since they used --download-p4pdes and 
> >> thus 
> >>have the fork on their system
> >>  they put the fix in new branch in the p4pdes fork and push it
> >>  they edit p4pdes.py and put a new commit in it pointing to 
> >> their branch in the fork
> >> run the pipeline again
> >> if fails with p4pdes they do the above again
> >> else the PETSc branch gets accepted and merged to master
> >>depending on Ed's choice we make an MR for p4pdes depending on 
> >> the agre

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Barry Smith]

  Model: When working with PETSc master.

The tests in a "master"  (not necessarily the exact master that the 
external person is using) of the external package run in the CI.

Goal: The PETSc developer can debug and make changes to PETSc and to the 
external package to fix the programs in the most painless way possible. With 
the least number of steps, MR, and generally forgetting things.

Example:  I change some PETSc code and the pipeline comes back with a 
failure on a p4pdes example.

I debug the crashing p4pdes example and fix a problem in it.

I now want to make a PETSc MR with my new PETSc code and update the copy of 
the external "master" so when my MR get's merged the pipelines will work with 
the external "master". 

  Other Goal:  The developer of the external package can update their code 
asynchronously with our work, that is if they delay we can keep on working on 
stuff on and on and we can update our test version asynchronously from their 
work. 

  I proposed a plan with --download-xxx which we already are comfortable with 
and understand. Jed proposed a plan with git tree we are not yet completely 
familiar with.

  Barry




> On Nov 1, 2020, at 8:56 PM, Satish Balay  wrote:
> 
> On Sun, 1 Nov 2020, Barry Smith wrote:
> 
>> 
>> 
>>> On Nov 1, 2020, at 2:57 PM, Jed Brown  wrote:
>>> 
>>> Barry Smith  writes:
>>> 
>>  I vote to fix things in our fork or gittree thing continuously since it 
>> makes it easier to fix things rather than wait to the release when we 
>> try to find and fix everything and it also helps tell us if we 
>> introduced a real bug into PETSc and fix PETSc immediately instead of 
>> waiting up to 6 months, just like we now we test immediately with 
>> Petsc4py and we should do with SLEPc.  How often we give the updates to 
>> Ed is a completely different issue. 
>> 
>> So again back to my original statement ,it comes down to if the subtree 
>> or the fork approach is easier for all the PETSc developers who do not 
>> currently know gittree and would need to learn it with your approach. I 
>> don't know which is easier learning to use gittree which has its own 
>> gotcha's or using mine which we all know but may require an extra step 
>> (not involving Ed, just updating the p4pdes.py commit each time we 
>> change something in the fork.)
>> 
>> I think using --download-p4pdes on a couple of systems in the CI is 
>> enough, I don't think we need to put it all CI pipelines (I would like 
>> slepc in all pipelines).but we could put in all pipelines if we want.
>> 
>> For completeness I show the exact the work flow for my suggestion
>> 
>> pipelines --download-p4pdes and runs its tests
>> if it breaks the developer uses --download-p4pdes  on their system 
>>   they fix the problem either by fixing PETSc or what is downloaded from 
>> the p4pde fork
>>  if the fix is in the p4pdes fork they make a branch in the p4pdes 
>> fork, which they already have since they used --download-p4pdes and thus 
>>have the fork on their system
>>  they put the fix in new branch in the p4pdes fork and push it
>>  they edit p4pdes.py and put a new commit in it pointing to 
>> their branch in the fork
>> run the pipeline again
>> if fails with p4pdes they do the above again
>> else the PETSc branch gets accepted and merged to master
>>depending on Ed's choice we make an MR for p4pdes depending on 
>> the agreed upon cycle. If Ed puts the fix into his master then we just 
>> update
>> our fork with his latest master with a simple merge of his.  
>> This will then become the new one we test against. If Ed doesn't respond 
>> to the MR all is fine we 
>>just continue on our fork. If he puts other things in his 
>> branch but not our MR we just merge that into our fork and so are still 
>> testing with his latest master.
> 
> As you've enumerated here, this requires three MRs per change:
 
 You still don't understand my suggested approach.
 
> 
> 1. in PETSc with the actual change and to point at a p4pdes commit
 
   Yes
 
> 2. in p4pdes (Ed's or our fork) to implement needed changes (note: this 
> can't merge until #1 merges)
 
   Not a MR just a new branch (or commit) in the fork off the last branch 
 and the commit that goes into petsc4py from the new branch.
 This is our fork of p4pdes and there is no reason to do MR on it. The MRs 
 are done by Ed based on a schedule Ed picks from here to his repo, if he 
 doesn't like something we've done he requests changes to MR branch as 
 always.
 
> 3. in PETSc to point at the merge commit of p4pdes (after #2 merges)
 
   No, the MR 1 already has the correct new commit the developer 

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Satish Balay via petsc-dev]

On Sun, 1 Nov 2020, Satish Balay via petsc-dev wrote:

> On Sun, 1 Nov 2020, Barry Smith wrote:
> 
> > 
> > 
> > > On Nov 1, 2020, at 2:57 PM, Jed Brown  wrote:
> > > 
> > > Barry Smith  writes:
> > > 
> >    I vote to fix things in our fork or gittree thing continuously since 
> >  it makes it easier to fix things rather than wait to the release when 
> >  we try to find and fix everything and it also helps tell us if we 
> >  introduced a real bug into PETSc and fix PETSc immediately instead of 
> >  waiting up to 6 months, just like we now we test immediately with 
> >  Petsc4py and we should do with SLEPc.  How often we give the updates 
> >  to Ed is a completely different issue. 
> >  
> >   So again back to my original statement ,it comes down to if the 
> >  subtree or the fork approach is easier for all the PETSc developers 
> >  who do not currently know gittree and would need to learn it with your 
> >  approach. I don't know which is easier learning to use gittree which 
> >  has its own gotcha's or using mine which we all know but may require 
> >  an extra step (not involving Ed, just updating the p4pdes.py commit 
> >  each time we change something in the fork.)
> >  
> >  I think using --download-p4pdes on a couple of systems in the CI is 
> >  enough, I don't think we need to put it all CI pipelines (I would like 
> >  slepc in all pipelines).but we could put in all pipelines if we want.
> >  
> >  For completeness I show the exact the work flow for my suggestion
> >  
> >   pipelines --download-p4pdes and runs its tests
> >   if it breaks the developer uses --download-p4pdes  on their system 
> > they fix the problem either by fixing PETSc or what is downloaded 
> >  from the p4pde fork
> >    if the fix is in the p4pdes fork they make a branch in the 
> >  p4pdes fork, which they already have since they used --download-p4pdes 
> >  and thus 
> >  have the fork on their system
> >    they put the fix in new branch in the p4pdes fork and push it
> >    they edit p4pdes.py and put a new commit in it pointing to 
> >  their branch in the fork
> >   run the pipeline again
> >   if fails with p4pdes they do the above again
> >   else the PETSc branch gets accepted and merged to master
> >  depending on Ed's choice we make an MR for p4pdes depending on 
> >  the agreed upon cycle. If Ed puts the fix into his master then we just 
> >  update
> >   our fork with his latest master with a simple merge of 
> >  his.  This will then become the new one we test against. If Ed doesn't 
> >  respond to the MR all is fine we 
> >  just continue on our fork. If he puts other things in his 
> >  branch but not our MR we just merge that into our fork and so are 
> >  still testing with his latest master.
> > >>> 
> > >>> As you've enumerated here, this requires three MRs per change:
> > >> 
> > >>  You still don't understand my suggested approach.
> > >> 
> > >>> 
> > >>> 1. in PETSc with the actual change and to point at a p4pdes commit
> > >> 
> > >>Yes
> > >> 
> > >>> 2. in p4pdes (Ed's or our fork) to implement needed changes (note: this 
> > >>> can't merge until #1 merges)
> > >> 
> > >>Not a MR just a new branch (or commit) in the fork off the last 
> > >> branch and the commit that goes into petsc4py from the new branch.
> > >>  This is our fork of p4pdes and there is no reason to do MR on it. The 
> > >> MRs are done by Ed based on a schedule Ed picks from here to his repo, 
> > >> if he doesn't like something we've done he requests changes to MR branch 
> > >> as always.
> > >> 
> > >>> 3. in PETSc to point at the merge commit of p4pdes (after #2 merges)
> > >> 
> > >>No, the MR 1 already has the correct new commit the developer put in 
> > >> step one. 
> > > 
> > > p4pdes.py either needs a branch name or a commit, but in practice, it 
> > > needs to be a commit. Why?
> > > 
> > >  petsc/master works with p4pdes/master
> > >  petsc/breaking-change works with p4pdes/breaking-change
> > > 
> > > We can't merge p4pdes/breaking-change to p4pdes/master until 
> > > petsc/breaking-change merges to petsc/master (otherwise it would break 
> > > everyone else's workflow). But now petsc/master will point at 
> > > p4pdes/breaking-change and needs to be updated after the merge.
> > > 
> > > Meanwhile, petsc/bug-fix branched off petsc/master prior to this ordeal 
> > > and still points at p4pdes/master, which will break after all the merges 
> > > occur. I guess you could stop the world and tell everyone to rebase or 
> > > merge from master, but that's even more disruptive.
> > > 
> > > So p4pdes.py needs to name a commit, not a branch.  You said:
> > > 
> >  depending on Ed's choice we make an MR for p4pdes depending

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Satish Balay via petsc-dev]

On Sun, 1 Nov 2020, Barry Smith wrote:

> 
> 
> > On Nov 1, 2020, at 2:57 PM, Jed Brown  wrote:
> > 
> > Barry Smith  writes:
> > 
>    I vote to fix things in our fork or gittree thing continuously since 
>  it makes it easier to fix things rather than wait to the release when we 
>  try to find and fix everything and it also helps tell us if we 
>  introduced a real bug into PETSc and fix PETSc immediately instead of 
>  waiting up to 6 months, just like we now we test immediately with 
>  Petsc4py and we should do with SLEPc.  How often we give the updates to 
>  Ed is a completely different issue. 
>  
>   So again back to my original statement ,it comes down to if the subtree 
>  or the fork approach is easier for all the PETSc developers who do not 
>  currently know gittree and would need to learn it with your approach. I 
>  don't know which is easier learning to use gittree which has its own 
>  gotcha's or using mine which we all know but may require an extra step 
>  (not involving Ed, just updating the p4pdes.py commit each time we 
>  change something in the fork.)
>  
>  I think using --download-p4pdes on a couple of systems in the CI is 
>  enough, I don't think we need to put it all CI pipelines (I would like 
>  slepc in all pipelines).but we could put in all pipelines if we want.
>  
>  For completeness I show the exact the work flow for my suggestion
>  
>   pipelines --download-p4pdes and runs its tests
>   if it breaks the developer uses --download-p4pdes  on their system 
> they fix the problem either by fixing PETSc or what is downloaded 
>  from the p4pde fork
>    if the fix is in the p4pdes fork they make a branch in the p4pdes 
>  fork, which they already have since they used --download-p4pdes and thus 
>  have the fork on their system
>    they put the fix in new branch in the p4pdes fork and push it
>    they edit p4pdes.py and put a new commit in it pointing to 
>  their branch in the fork
>   run the pipeline again
>   if fails with p4pdes they do the above again
>   else the PETSc branch gets accepted and merged to master
>  depending on Ed's choice we make an MR for p4pdes depending on 
>  the agreed upon cycle. If Ed puts the fix into his master then we just 
>  update
>   our fork with his latest master with a simple merge of his. 
>   This will then become the new one we test against. If Ed doesn't 
>  respond to the MR all is fine we 
>  just continue on our fork. If he puts other things in his 
>  branch but not our MR we just merge that into our fork and so are still 
>  testing with his latest master.
> >>> 
> >>> As you've enumerated here, this requires three MRs per change:
> >> 
> >>  You still don't understand my suggested approach.
> >> 
> >>> 
> >>> 1. in PETSc with the actual change and to point at a p4pdes commit
> >> 
> >>Yes
> >> 
> >>> 2. in p4pdes (Ed's or our fork) to implement needed changes (note: this 
> >>> can't merge until #1 merges)
> >> 
> >>Not a MR just a new branch (or commit) in the fork off the last branch 
> >> and the commit that goes into petsc4py from the new branch.
> >>  This is our fork of p4pdes and there is no reason to do MR on it. The MRs 
> >> are done by Ed based on a schedule Ed picks from here to his repo, if he 
> >> doesn't like something we've done he requests changes to MR branch as 
> >> always.
> >> 
> >>> 3. in PETSc to point at the merge commit of p4pdes (after #2 merges)
> >> 
> >>No, the MR 1 already has the correct new commit the developer put in 
> >> step one. 
> > 
> > p4pdes.py either needs a branch name or a commit, but in practice, it needs 
> > to be a commit. Why?
> > 
> >  petsc/master works with p4pdes/master
> >  petsc/breaking-change works with p4pdes/breaking-change
> > 
> > We can't merge p4pdes/breaking-change to p4pdes/master until 
> > petsc/breaking-change merges to petsc/master (otherwise it would break 
> > everyone else's workflow). But now petsc/master will point at 
> > p4pdes/breaking-change and needs to be updated after the merge.
> > 
> > Meanwhile, petsc/bug-fix branched off petsc/master prior to this ordeal and 
> > still points at p4pdes/master, which will break after all the merges occur. 
> > I guess you could stop the world and tell everyone to rebase or merge from 
> > master, but that's even more disruptive.
> > 
> > So p4pdes.py needs to name a commit, not a branch.  You said:
> > 
>  depending on Ed's choice we make an MR for p4pdes depending on 
>  the agreed upon cycle. If Ed puts the fix into his master then we just 
>  update
>   our fork with his latest master with a simple merge of his. 
>   This will then become the new one we test against.
> > 
> > After any 

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Barry Smith]

> On Nov 1, 2020, at 2:57 PM, Jed Brown  wrote:
> 
> Barry Smith  writes:
> 
   I vote to fix things in our fork or gittree thing continuously since it 
 makes it easier to fix things rather than wait to the release when we try 
 to find and fix everything and it also helps tell us if we introduced a 
 real bug into PETSc and fix PETSc immediately instead of waiting up to 6 
 months, just like we now we test immediately with Petsc4py and we should 
 do with SLEPc.  How often we give the updates to Ed is a completely 
 different issue. 
 
  So again back to my original statement ,it comes down to if the subtree 
 or the fork approach is easier for all the PETSc developers who do not 
 currently know gittree and would need to learn it with your approach. I 
 don't know which is easier learning to use gittree which has its own 
 gotcha's or using mine which we all know but may require an extra step 
 (not involving Ed, just updating the p4pdes.py commit each time we change 
 something in the fork.)
 
 I think using --download-p4pdes on a couple of systems in the CI is 
 enough, I don't think we need to put it all CI pipelines (I would like 
 slepc in all pipelines).but we could put in all pipelines if we want.
 
 For completeness I show the exact the work flow for my suggestion
 
  pipelines --download-p4pdes and runs its tests
  if it breaks the developer uses --download-p4pdes  on their system 
they fix the problem either by fixing PETSc or what is downloaded from 
 the p4pde fork
   if the fix is in the p4pdes fork they make a branch in the p4pdes 
 fork, which they already have since they used --download-p4pdes and thus 
 have the fork on their system
   they put the fix in new branch in the p4pdes fork and push it
   they edit p4pdes.py and put a new commit in it pointing to their 
 branch in the fork
  run the pipeline again
  if fails with p4pdes they do the above again
  else the PETSc branch gets accepted and merged to master
 depending on Ed's choice we make an MR for p4pdes depending on the 
 agreed upon cycle. If Ed puts the fix into his master then we just update
  our fork with his latest master with a simple merge of his.  
 This will then become the new one we test against. If Ed doesn't respond 
 to the MR all is fine we 
 just continue on our fork. If he puts other things in his 
 branch but not our MR we just merge that into our fork and so are still 
 testing with his latest master.
>>> 
>>> As you've enumerated here, this requires three MRs per change:
>> 
>>  You still don't understand my suggested approach.
>> 
>>> 
>>> 1. in PETSc with the actual change and to point at a p4pdes commit
>> 
>>Yes
>> 
>>> 2. in p4pdes (Ed's or our fork) to implement needed changes (note: this 
>>> can't merge until #1 merges)
>> 
>>Not a MR just a new branch (or commit) in the fork off the last branch 
>> and the commit that goes into petsc4py from the new branch.
>>  This is our fork of p4pdes and there is no reason to do MR on it. The MRs 
>> are done by Ed based on a schedule Ed picks from here to his repo, if he 
>> doesn't like something we've done he requests changes to MR branch as always.
>> 
>>> 3. in PETSc to point at the merge commit of p4pdes (after #2 merges)
>> 
>>No, the MR 1 already has the correct new commit the developer put in step 
>> one. 
> 
> p4pdes.py either needs a branch name or a commit, but in practice, it needs 
> to be a commit. Why?
> 
>  petsc/master works with p4pdes/master
>  petsc/breaking-change works with p4pdes/breaking-change
> 
> We can't merge p4pdes/breaking-change to p4pdes/master until 
> petsc/breaking-change merges to petsc/master (otherwise it would break 
> everyone else's workflow). But now petsc/master will point at 
> p4pdes/breaking-change and needs to be updated after the merge.
> 
> Meanwhile, petsc/bug-fix branched off petsc/master prior to this ordeal and 
> still points at p4pdes/master, which will break after all the merges occur. I 
> guess you could stop the world and tell everyone to rebase or merge from 
> master, but that's even more disruptive.
> 
> So p4pdes.py needs to name a commit, not a branch.  You said:
> 
 depending on Ed's choice we make an MR for p4pdes depending on the 
 agreed upon cycle. If Ed puts the fix into his master then we just update
  our fork with his latest master with a simple merge of his.  
 This will then become the new one we test against.
> 
> After any such merge, we need to update p4pdes.py with the merge commit.
> 
>> 
>>  For p4pdes release we can do the same thing but make sure for each change 
>> we make an MR for Ed immediately. 
>> 
>>> 
>>> With subtree, there is only one MR and it's in the PETS

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Jed Brown]

Barry Smith  writes:

>>>I vote to fix things in our fork or gittree thing continuously since it 
>>> makes it easier to fix things rather than wait to the release when we try 
>>> to find and fix everything and it also helps tell us if we introduced a 
>>> real bug into PETSc and fix PETSc immediately instead of waiting up to 6 
>>> months, just like we now we test immediately with Petsc4py and we should do 
>>> with SLEPc.  How often we give the updates to Ed is a completely different 
>>> issue. 
>>> 
>>>   So again back to my original statement ,it comes down to if the subtree 
>>> or the fork approach is easier for all the PETSc developers who do not 
>>> currently know gittree and would need to learn it with your approach. I 
>>> don't know which is easier learning to use gittree which has its own 
>>> gotcha's or using mine which we all know but may require an extra step (not 
>>> involving Ed, just updating the p4pdes.py commit each time we change 
>>> something in the fork.)
>>> 
>>>  I think using --download-p4pdes on a couple of systems in the CI is 
>>> enough, I don't think we need to put it all CI pipelines (I would like 
>>> slepc in all pipelines).but we could put in all pipelines if we want.
>>> 
>>>  For completeness I show the exact the work flow for my suggestion
>>> 
>>>   pipelines --download-p4pdes and runs its tests
>>>   if it breaks the developer uses --download-p4pdes  on their system 
>>> they fix the problem either by fixing PETSc or what is downloaded from 
>>> the p4pde fork
>>>if the fix is in the p4pdes fork they make a branch in the p4pdes 
>>> fork, which they already have since they used --download-p4pdes and thus 
>>>  have the fork on their system
>>>they put the fix in new branch in the p4pdes fork and push it
>>>they edit p4pdes.py and put a new commit in it pointing to their 
>>> branch in the fork
>>>   run the pipeline again
>>>   if fails with p4pdes they do the above again
>>>   else the PETSc branch gets accepted and merged to master
>>>  depending on Ed's choice we make an MR for p4pdes depending on the 
>>> agreed upon cycle. If Ed puts the fix into his master then we just update
>>>   our fork with his latest master with a simple merge of his.  
>>> This will then become the new one we test against. If Ed doesn't respond to 
>>> the MR all is fine we 
>>>  just continue on our fork. If he puts other things in his 
>>> branch but not our MR we just merge that into our fork and so are still 
>>> testing with his latest master.
>> 
>> As you've enumerated here, this requires three MRs per change:
>
>   You still don't understand my suggested approach.
>
>> 
>>  1. in PETSc with the actual change and to point at a p4pdes commit
>
> Yes
>
>>  2. in p4pdes (Ed's or our fork) to implement needed changes (note: this 
>> can't merge until #1 merges)
>
> Not a MR just a new branch (or commit) in the fork off the last branch 
> and the commit that goes into petsc4py from the new branch.
>   This is our fork of p4pdes and there is no reason to do MR on it. The MRs 
> are done by Ed based on a schedule Ed picks from here to his repo, if he 
> doesn't like something we've done he requests changes to MR branch as always.
>
>>  3. in PETSc to point at the merge commit of p4pdes (after #2 merges)
>
> No, the MR 1 already has the correct new commit the developer put in step 
> one. 

p4pdes.py either needs a branch name or a commit, but in practice, it needs to 
be a commit. Why?

  petsc/master works with p4pdes/master
  petsc/breaking-change works with p4pdes/breaking-change

We can't merge p4pdes/breaking-change to p4pdes/master until 
petsc/breaking-change merges to petsc/master (otherwise it would break everyone 
else's workflow). But now petsc/master will point at p4pdes/breaking-change and 
needs to be updated after the merge.

Meanwhile, petsc/bug-fix branched off petsc/master prior to this ordeal and 
still points at p4pdes/master, which will break after all the merges occur. I 
guess you could stop the world and tell everyone to rebase or merge from 
master, but that's even more disruptive.

So p4pdes.py needs to name a commit, not a branch.  You said:

>>>  depending on Ed's choice we make an MR for p4pdes depending on the 
>>> agreed upon cycle. If Ed puts the fix into his master then we just update
>>>   our fork with his latest master with a simple merge of his.  
>>> This will then become the new one we test against.

After any such merge, we need to update p4pdes.py with the merge commit.

>
>   For p4pdes release we can do the same thing but make sure for each change 
> we make an MR for Ed immediately. 
>
>> 
>> With subtree, there is only one MR and it's in the PETSc repository. 
>
>   Please show the whole work flow with gittree

Everything is in the PETSc repository and developers don't need to know about 
the p4pdes 

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Satish Balay via petsc-dev]

BTW: wrt fork (clone) vs subtree  (I'll have to read 
https://medium.com/@porteneuve/mastering-git-subtrees-943d29a798ec again) - the 
way I understand it is:

One can do push/pull/merge with subtree that are equivalent to push/pull/merge 
with a clone.

i.e; with a local clone one does: [branch is implicit here so lets stick with 
default master branch]

git clone repo-URL
cd repo

git commit
git pull
git push

with subtree one does:

cd petsc
git subtree add --prefix=location repo-url [branch]

git commit
git subtree pull --prefix=location repo-url [branch]
git subtree push --prefix=location repo-url [branch]

i.e the complexity of branches/merge issues don't go away - it just avoids the 
extra repo [and avoids making 2 commits in sync in 2 different repos]

Notes:
- wrt petsc4py we don't have a 'subtree pull' in the workflow
- and 'subtree push' is currently broken - so petsc4py workflow is currently 
broken.

Also note: with the previous petsc4py workflow we had:
 - make MR to petsc4py
 - use this commit with --download-petsc4py in petsc [in MR branch]
 - do not merge petsc-MR until petsc4py-MR is merged [this way - any revisions 
in petsc4py-MR - that will change the hash can be updated in petsc-MR]

With the current discussion - the workflow for examples is more complicated - 
i.e the changes petsc-developers do and that get accepted upstream can be 
different [perhaps temporarily.] - so there is extra complexity here where 
local changes (fork or subrepo) are not in sync with upstream changes. One way 
to sync is:
- make local changes (and use them with petsc CI)
- make separate MR to upstream (revise and update the MR - as needed for 
acceptance). Note: I'm not sure if there is a workflow to test these revisions 
in petsc cI
- once merged to upstream example repo - sync [subtree, or fork] - either a 
merge - or revert local changes, pull upstream changes.

So a simpler model is [as Ed referred to]: use upstream only, and the CI use 
(allow_failure:true) - and then Ed can fix the issues as he sees fit [and there 
will be CI errors until the fix is in]. And the petsc-MR is not held back (even 
with fundamental failures)

Satish

On Sun, 1 Nov 2020, Satish Balay wrote:

> I should add - we should:
> 
> - ignore git subtree or a fork
> - assume one (or more) petsc developers have write access to create branches 
> in upstream repo
> - figure out workflow for the requirements [and current constraints wrt book].
> 
> - a fork primarily helps with write access [wrt branches, when no write 
> access is available for upstream repo] - there is a mirror sync cost
> - a subtree [assuming its not buggy] - helps with a single commit workflow 
> [vs multiple commits in different repos - and keeping them in sync]
> - [a default is neither - using upstream repo directly]
> 
> And in all cases there is similar synchronization cost between upstream repo 
> and fork/subtree [this depends on the requirements - and current constranits]
> 
> Satish
> 
> On Sun, 1 Nov 2020, Satish Balay via petsc-dev wrote:
> 
> > On Sun, 1 Nov 2020, Barry Smith wrote:
> > 
> > > 
> > > 
> > > > On Nov 1, 2020, at 10:30 AM, Satish Balay  wrote:
> > > > 
> > > > Just a note: subtree workflow is currently broken with petsc4py
> > > > 
> > > > We've imported petsc4py using subtree - but now we are unable to export 
> > > > back changes in petsc repo to the standalone petsc4py repo due to git 
> > > > subtree errors.
> > > > 
> > > > I spent some time on workarounds - but didn't go anywhere.
> > > > 
> > > > [likely this is a non-issue with this examples repo workflow - as there 
> > > > is no direct push back to upstream repo]
> > > > 
> > > > And with subtree - not all developers need to know about it. Only the 
> > > > person who is syncing the 2 repos.. [if they know how to sync/resolve 
> > > > conflicts]
> > > 
> > >   Since each person is reponsable for their branch getting merged, 
> > > doesn't this mean anyone who changes something in PETSc that requires a 
> > > change in p4pdes has to fix it via gittree process ( that is have have to 
> > > sync the 2 repose?) and hence now how to sync 2 repos via gittree.
> > 
> > 
> > For one its not clear what problem is being addressed here [as mentioned in 
> > my other e-mail.]
> > 
> > >From Ed's e-mail - my understanding is - whatever changes petsc developers 
> > >might have - might not be acceptable.
> > 
> > So I think first we need to figure out what is acceptable and not - and 
> > then figure out the workflow.
> > 
> > " Since each person is reponsable for their branch getting merged, "
> > 
> > So its not clear to me if this fits with what Ed mentioned. [wrt his goals 
> > of keeping the examples in sync with the book]
> > 
> > BTW I should note: the current process [which I don't understand 
> > completely] looks more complicated than what we had with petsc4py - which 
> > you were against at - and raised this at  every opportunity - until 
> > petsc4py was imported into petsc (usi

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Satish Balay via petsc-dev]

I should add - we should:

- ignore git subtree or a fork
- assume one (or more) petsc developers have write access to create branches in 
upstream repo
- figure out workflow for the requirements [and current constraints wrt book].

- a fork primarily helps with write access [wrt branches, when no write access 
is available for upstream repo] - there is a mirror sync cost
- a subtree [assuming its not buggy] - helps with a single commit workflow [vs 
multiple commits in different repos - and keeping them in sync]
- [a default is neither - using upstream repo directly]

And in all cases there is similar synchronization cost between upstream repo 
and fork/subtree [this depends on the requirements - and current constranits]

Satish

On Sun, 1 Nov 2020, Satish Balay via petsc-dev wrote:

> On Sun, 1 Nov 2020, Barry Smith wrote:
> 
> > 
> > 
> > > On Nov 1, 2020, at 10:30 AM, Satish Balay  wrote:
> > > 
> > > Just a note: subtree workflow is currently broken with petsc4py
> > > 
> > > We've imported petsc4py using subtree - but now we are unable to export 
> > > back changes in petsc repo to the standalone petsc4py repo due to git 
> > > subtree errors.
> > > 
> > > I spent some time on workarounds - but didn't go anywhere.
> > > 
> > > [likely this is a non-issue with this examples repo workflow - as there 
> > > is no direct push back to upstream repo]
> > > 
> > > And with subtree - not all developers need to know about it. Only the 
> > > person who is syncing the 2 repos.. [if they know how to sync/resolve 
> > > conflicts]
> > 
> >   Since each person is reponsable for their branch getting merged, doesn't 
> > this mean anyone who changes something in PETSc that requires a change in 
> > p4pdes has to fix it via gittree process ( that is have have to sync the 2 
> > repose?) and hence now how to sync 2 repos via gittree.
> 
> 
> For one its not clear what problem is being addressed here [as mentioned in 
> my other e-mail.]
> 
> >From Ed's e-mail - my understanding is - whatever changes petsc developers 
> >might have - might not be acceptable.
> 
> So I think first we need to figure out what is acceptable and not - and then 
> figure out the workflow.
> 
> " Since each person is reponsable for their branch getting merged, "
> 
> So its not clear to me if this fits with what Ed mentioned. [wrt his goals of 
> keeping the examples in sync with the book]
> 
> BTW I should note: the current process [which I don't understand completely] 
> looks more complicated than what we had with petsc4py - which you were 
> against at - and raised this at  every opportunity - until petsc4py was 
> imported into petsc (using subtree)
> 
> Satish
> 
> > 
> >   Until I see the entire gittree workflow I will not understand.
> > 
> >   Barry
> > 
> > > 
> > > Satish
> > > 
> > > On Sun, 1 Nov 2020, Jed Brown wrote:
> > > 
> > >> Barry Smith  writes:
> > >> 
> > > git subtree does require use of new commands every time you mess with 
> > > it (say every three months) that we do not know and since each of 
> > > us will do this infrequently it is likely we will not remember them 
> > > (I won't)  while my approach does not require remembering new 
> > > commands. 
> > > My approach only requires branching, push, and pulling on the fork 
> > > and updating EdsRepo.py which is things all the developers know about 
> > > and 
> > > do regularly since we update other external packages.
> > >>> 
> > > Circular dependencies (PETSc CI depends on Ed's p4pdes depends on 
> > > PETSc) are significantly more labor-intensive and it would need to be 
> > > done on each change, versus once per release cycle.
> > >>> 
> > >>>I think we are not communicating on exactly the  same wave length. 
> > >>> 
> > >>>I am not advocating we test on Ed's I am advocating we test on our 
> > >>> fork of Ed's and how often our fork gets passed to Ed with MR is a 
> > >>> choice all of us make together, now we could do it once at release time 
> > >>> with all the fixes we made, every month or each time. Totally up to Ed 
> > >>> how often he wants to get them, and he is free to ignore them for weeks 
> > >>> (up to the next release if he likes) since our testing will continue 
> > >>> because we use our fork. How often we pass them on to Ed is not related 
> > >>> to how we maintain our CI.
> > >>> 
> > >>>There is no circular dependency on my approach with Ed's p4pdes.
> > >> 
> > >> The circular dependency is on our fork of p4pdes.
> > >> 
> > >>>I vote to fix things in our fork or gittree thing continuously since 
> > >>> it makes it easier to fix things rather than wait to the release when 
> > >>> we try to find and fix everything and it also helps tell us if we 
> > >>> introduced a real bug into PETSc and fix PETSc immediately instead of 
> > >>> waiting up to 6 months, just like we now we test immediately with 
> > >>> Petsc4py and we should do with SLEPc.  How often we give the update

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Satish Balay via petsc-dev]

On Sun, 1 Nov 2020, Barry Smith wrote:

> 
> 
> > On Nov 1, 2020, at 10:30 AM, Satish Balay  wrote:
> > 
> > Just a note: subtree workflow is currently broken with petsc4py
> > 
> > We've imported petsc4py using subtree - but now we are unable to export 
> > back changes in petsc repo to the standalone petsc4py repo due to git 
> > subtree errors.
> > 
> > I spent some time on workarounds - but didn't go anywhere.
> > 
> > [likely this is a non-issue with this examples repo workflow - as there is 
> > no direct push back to upstream repo]
> > 
> > And with subtree - not all developers need to know about it. Only the 
> > person who is syncing the 2 repos.. [if they know how to sync/resolve 
> > conflicts]
> 
>   Since each person is reponsable for their branch getting merged, doesn't 
> this mean anyone who changes something in PETSc that requires a change in 
> p4pdes has to fix it via gittree process ( that is have have to sync the 2 
> repose?) and hence now how to sync 2 repos via gittree.


For one its not clear what problem is being addressed here [as mentioned in my 
other e-mail.]

>From Ed's e-mail - my understanding is - whatever changes petsc developers 
>might have - might not be acceptable.

So I think first we need to figure out what is acceptable and not - and then 
figure out the workflow.

" Since each person is reponsable for their branch getting merged, "

So its not clear to me if this fits with what Ed mentioned. [wrt his goals of 
keeping the examples in sync with the book]

BTW I should note: the current process [which I don't understand completely] 
looks more complicated than what we had with petsc4py - which you were against 
at - and raised this at  every opportunity - until petsc4py was imported into 
petsc (using subtree)

Satish

> 
>   Until I see the entire gittree workflow I will not understand.
> 
>   Barry
> 
> > 
> > Satish
> > 
> > On Sun, 1 Nov 2020, Jed Brown wrote:
> > 
> >> Barry Smith  writes:
> >> 
> > git subtree does require use of new commands every time you mess with 
> > it (say every three months) that we do not know and since each of 
> > us will do this infrequently it is likely we will not remember them (I 
> > won't)  while my approach does not require remembering new commands. 
> > My approach only requires branching, push, and pulling on the fork and 
> > updating EdsRepo.py which is things all the developers know about and 
> > do regularly since we update other external packages.
> >>> 
> > Circular dependencies (PETSc CI depends on Ed's p4pdes depends on 
> > PETSc) are significantly more labor-intensive and it would need to be 
> > done on each change, versus once per release cycle.
> >>> 
> >>>I think we are not communicating on exactly the  same wave length. 
> >>> 
> >>>I am not advocating we test on Ed's I am advocating we test on our 
> >>> fork of Ed's and how often our fork gets passed to Ed with MR is a choice 
> >>> all of us make together, now we could do it once at release time with all 
> >>> the fixes we made, every month or each time. Totally up to Ed how often 
> >>> he wants to get them, and he is free to ignore them for weeks (up to the 
> >>> next release if he likes) since our testing will continue because we use 
> >>> our fork. How often we pass them on to Ed is not related to how we 
> >>> maintain our CI.
> >>> 
> >>>There is no circular dependency on my approach with Ed's p4pdes.
> >> 
> >> The circular dependency is on our fork of p4pdes.
> >> 
> >>>I vote to fix things in our fork or gittree thing continuously since 
> >>> it makes it easier to fix things rather than wait to the release when we 
> >>> try to find and fix everything and it also helps tell us if we introduced 
> >>> a real bug into PETSc and fix PETSc immediately instead of waiting up to 
> >>> 6 months, just like we now we test immediately with Petsc4py and we 
> >>> should do with SLEPc.  How often we give the updates to Ed is a 
> >>> completely different issue. 
> >>> 
> >>>   So again back to my original statement ,it comes down to if the subtree 
> >>> or the fork approach is easier for all the PETSc developers who do not 
> >>> currently know gittree and would need to learn it with your approach. I 
> >>> don't know which is easier learning to use gittree which has its own 
> >>> gotcha's or using mine which we all know but may require an extra step 
> >>> (not involving Ed, just updating the p4pdes.py commit each time we change 
> >>> something in the fork.)
> >>> 
> >>>  I think using --download-p4pdes on a couple of systems in the CI is 
> >>> enough, I don't think we need to put it all CI pipelines (I would like 
> >>> slepc in all pipelines).but we could put in all pipelines if we want.
> >>> 
> >>>  For completeness I show the exact the work flow for my suggestion
> >>> 
> >>>   pipelines --download-p4pdes and runs its tests
> >>>   if it breaks the developer uses --download-p4pdes  on thei

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Barry Smith]

> On Nov 1, 2020, at 10:30 AM, Satish Balay  wrote:
> 
> Just a note: subtree workflow is currently broken with petsc4py
> 
> We've imported petsc4py using subtree - but now we are unable to export back 
> changes in petsc repo to the standalone petsc4py repo due to git subtree 
> errors.
> 
> I spent some time on workarounds - but didn't go anywhere.
> 
> [likely this is a non-issue with this examples repo workflow - as there is no 
> direct push back to upstream repo]
> 
> And with subtree - not all developers need to know about it. Only the person 
> who is syncing the 2 repos.. [if they know how to sync/resolve conflicts]

  Since each person is reponsable for their branch getting merged, doesn't this 
mean anyone who changes something in PETSc that requires a change in p4pdes has 
to fix it via gittree process ( that is have have to sync the 2 repose?) and 
hence now how to sync 2 repos via gittree. 

  Until I see the entire gittree workflow I will not understand.

  Barry

> 
> Satish
> 
> On Sun, 1 Nov 2020, Jed Brown wrote:
> 
>> Barry Smith  writes:
>> 
> git subtree does require use of new commands every time you mess with it 
> (say every three months) that we do not know and since each of 
> us will do this infrequently it is likely we will not remember them (I 
> won't)  while my approach does not require remembering new commands. 
> My approach only requires branching, push, and pulling on the fork and 
> updating EdsRepo.py which is things all the developers know about and 
> do regularly since we update other external packages.
>>> 
> Circular dependencies (PETSc CI depends on Ed's p4pdes depends on PETSc) 
> are significantly more labor-intensive and it would need to be done on 
> each change, versus once per release cycle.
>>> 
>>>I think we are not communicating on exactly the  same wave length. 
>>> 
>>>I am not advocating we test on Ed's I am advocating we test on our fork 
>>> of Ed's and how often our fork gets passed to Ed with MR is a choice all of 
>>> us make together, now we could do it once at release time with all the 
>>> fixes we made, every month or each time. Totally up to Ed how often he 
>>> wants to get them, and he is free to ignore them for weeks (up to the next 
>>> release if he likes) since our testing will continue because we use our 
>>> fork. How often we pass them on to Ed is not related to how we maintain our 
>>> CI.
>>> 
>>>There is no circular dependency on my approach with Ed's p4pdes.
>> 
>> The circular dependency is on our fork of p4pdes.
>> 
>>>I vote to fix things in our fork or gittree thing continuously since it 
>>> makes it easier to fix things rather than wait to the release when we try 
>>> to find and fix everything and it also helps tell us if we introduced a 
>>> real bug into PETSc and fix PETSc immediately instead of waiting up to 6 
>>> months, just like we now we test immediately with Petsc4py and we should do 
>>> with SLEPc.  How often we give the updates to Ed is a completely different 
>>> issue. 
>>> 
>>>   So again back to my original statement ,it comes down to if the subtree 
>>> or the fork approach is easier for all the PETSc developers who do not 
>>> currently know gittree and would need to learn it with your approach. I 
>>> don't know which is easier learning to use gittree which has its own 
>>> gotcha's or using mine which we all know but may require an extra step (not 
>>> involving Ed, just updating the p4pdes.py commit each time we change 
>>> something in the fork.)
>>> 
>>>  I think using --download-p4pdes on a couple of systems in the CI is 
>>> enough, I don't think we need to put it all CI pipelines (I would like 
>>> slepc in all pipelines).but we could put in all pipelines if we want.
>>> 
>>>  For completeness I show the exact the work flow for my suggestion
>>> 
>>>   pipelines --download-p4pdes and runs its tests
>>>   if it breaks the developer uses --download-p4pdes  on their system 
>>> they fix the problem either by fixing PETSc or what is downloaded from 
>>> the p4pde fork
>>>if the fix is in the p4pdes fork they make a branch in the p4pdes 
>>> fork, which they already have since they used --download-p4pdes and thus 
>>>  have the fork on their system
>>>they put the fix in new branch in the p4pdes fork and push it
>>>they edit p4pdes.py and put a new commit in it pointing to their 
>>> branch in the fork
>>>   run the pipeline again
>>>   if fails with p4pdes they do the above again
>>>   else the PETSc branch gets accepted and merged to master
>>>  depending on Ed's choice we make an MR for p4pdes depending on the 
>>> agreed upon cycle. If Ed puts the fix into his master then we just update
>>>   our fork with his latest master with a simple merge of his.  
>>> This will then become the new one we test against. If Ed doesn't respond to 
>>> the MR all

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Barry Smith]

> On Nov 1, 2020, at 8:01 AM, Jed Brown  wrote:
> 
> Barry Smith  writes:
> 
 git subtree does require use of new commands every time you mess with it 
 (say every three months) that we do not know and since each of 
 us will do this infrequently it is likely we will not remember them (I 
 won't)  while my approach does not require remembering new commands. 
 My approach only requires branching, push, and pulling on the fork and 
 updating EdsRepo.py which is things all the developers know about and 
 do regularly since we update other external packages.
>> 
 Circular dependencies (PETSc CI depends on Ed's p4pdes depends on PETSc) 
 are significantly more labor-intensive and it would need to be done on 
 each change, versus once per release cycle.
>> 
>>I think we are not communicating on exactly the  same wave length. 
>> 
>>I am not advocating we test on Ed's I am advocating we test on our fork 
>> of Ed's and how often our fork gets passed to Ed with MR is a choice all of 
>> us make together, now we could do it once at release time with all the fixes 
>> we made, every month or each time. Totally up to Ed how often he wants to 
>> get them, and he is free to ignore them for weeks (up to the next release if 
>> he likes) since our testing will continue because we use our fork. How often 
>> we pass them on to Ed is not related to how we maintain our CI.
>> 
>>There is no circular dependency on my approach with Ed's p4pdes.
> 
> The circular dependency is on our fork of p4pdes.
> 
>>I vote to fix things in our fork or gittree thing continuously since it 
>> makes it easier to fix things rather than wait to the release when we try to 
>> find and fix everything and it also helps tell us if we introduced a real 
>> bug into PETSc and fix PETSc immediately instead of waiting up to 6 months, 
>> just like we now we test immediately with Petsc4py and we should do with 
>> SLEPc.  How often we give the updates to Ed is a completely different issue. 
>> 
>>   So again back to my original statement ,it comes down to if the subtree or 
>> the fork approach is easier for all the PETSc developers who do not 
>> currently know gittree and would need to learn it with your approach. I 
>> don't know which is easier learning to use gittree which has its own 
>> gotcha's or using mine which we all know but may require an extra step (not 
>> involving Ed, just updating the p4pdes.py commit each time we change 
>> something in the fork.)
>> 
>>  I think using --download-p4pdes on a couple of systems in the CI is enough, 
>> I don't think we need to put it all CI pipelines (I would like slepc in all 
>> pipelines).but we could put in all pipelines if we want.
>> 
>>  For completeness I show the exact the work flow for my suggestion
>> 
>>   pipelines --download-p4pdes and runs its tests
>>   if it breaks the developer uses --download-p4pdes  on their system 
>> they fix the problem either by fixing PETSc or what is downloaded from 
>> the p4pde fork
>>if the fix is in the p4pdes fork they make a branch in the p4pdes 
>> fork, which they already have since they used --download-p4pdes and thus 
>>  have the fork on their system
>>they put the fix in new branch in the p4pdes fork and push it
>>they edit p4pdes.py and put a new commit in it pointing to their 
>> branch in the fork
>>   run the pipeline again
>>   if fails with p4pdes they do the above again
>>   else the PETSc branch gets accepted and merged to master
>>  depending on Ed's choice we make an MR for p4pdes depending on the 
>> agreed upon cycle. If Ed puts the fix into his master then we just update
>>   our fork with his latest master with a simple merge of his.  
>> This will then become the new one we test against. If Ed doesn't respond to 
>> the MR all is fine we 
>>  just continue on our fork. If he puts other things in his 
>> branch but not our MR we just merge that into our fork and so are still 
>> testing with his latest master.
> 
> As you've enumerated here, this requires three MRs per change:

  You still don't understand my suggested approach.

> 
>  1. in PETSc with the actual change and to point at a p4pdes commit

Yes

>  2. in p4pdes (Ed's or our fork) to implement needed changes (note: this 
> can't merge until #1 merges)

Not a MR just a new branch (or commit) in the fork off the last branch and 
the commit that goes into petsc4py from the new branch.
  This is our fork of p4pdes and there is no reason to do MR on it. The MRs are 
done by Ed based on a schedule Ed picks from here to his repo, if he doesn't 
like something we've done he requests changes to MR branch as always.

>  3. in PETSc to point at the merge commit of p4pdes (after #2 merges)

No, the MR 1 already has the correct new commit the developer put in step 
one. 


  For p4pdes release we can do the same 

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Satish Balay via petsc-dev]

Just a note: subtree workflow is currently broken with petsc4py

We've imported petsc4py using subtree - but now we are unable to export back 
changes in petsc repo to the standalone petsc4py repo due to git subtree errors.

I spent some time on workarounds - but didn't go anywhere.

[likely this is a non-issue with this examples repo workflow - as there is no 
direct push back to upstream repo]

And with subtree - not all developers need to know about it. Only the person 
who is syncing the 2 repos.. [if they know how to sync/resolve conflicts]

Satish

On Sun, 1 Nov 2020, Jed Brown wrote:

> Barry Smith  writes:
> 
> >>> git subtree does require use of new commands every time you mess with it 
> >>> (say every three months) that we do not know and since each of 
> >>> us will do this infrequently it is likely we will not remember them (I 
> >>> won't)  while my approach does not require remembering new commands. 
> >>> My approach only requires branching, push, and pulling on the fork and 
> >>> updating EdsRepo.py which is things all the developers know about and 
> >>> do regularly since we update other external packages.
> >
> >>> Circular dependencies (PETSc CI depends on Ed's p4pdes depends on PETSc) 
> >>> are significantly more labor-intensive and it would need to be done on 
> >>> each change, versus once per release cycle.
> >
> > I think we are not communicating on exactly the  same wave length. 
> >
> > I am not advocating we test on Ed's I am advocating we test on our fork 
> > of Ed's and how often our fork gets passed to Ed with MR is a choice all of 
> > us make together, now we could do it once at release time with all the 
> > fixes we made, every month or each time. Totally up to Ed how often he 
> > wants to get them, and he is free to ignore them for weeks (up to the next 
> > release if he likes) since our testing will continue because we use our 
> > fork. How often we pass them on to Ed is not related to how we maintain our 
> > CI.
> >
> > There is no circular dependency on my approach with Ed's p4pdes.
> 
> The circular dependency is on our fork of p4pdes.
> 
> > I vote to fix things in our fork or gittree thing continuously since it 
> > makes it easier to fix things rather than wait to the release when we try 
> > to find and fix everything and it also helps tell us if we introduced a 
> > real bug into PETSc and fix PETSc immediately instead of waiting up to 6 
> > months, just like we now we test immediately with Petsc4py and we should do 
> > with SLEPc.  How often we give the updates to Ed is a completely different 
> > issue. 
> >
> >So again back to my original statement ,it comes down to if the subtree 
> > or the fork approach is easier for all the PETSc developers who do not 
> > currently know gittree and would need to learn it with your approach. I 
> > don't know which is easier learning to use gittree which has its own 
> > gotcha's or using mine which we all know but may require an extra step (not 
> > involving Ed, just updating the p4pdes.py commit each time we change 
> > something in the fork.)
> >
> >   I think using --download-p4pdes on a couple of systems in the CI is 
> > enough, I don't think we need to put it all CI pipelines (I would like 
> > slepc in all pipelines).but we could put in all pipelines if we want.
> >
> >   For completeness I show the exact the work flow for my suggestion
> >
> >pipelines --download-p4pdes and runs its tests
> >if it breaks the developer uses --download-p4pdes  on their system 
> >  they fix the problem either by fixing PETSc or what is downloaded from 
> > the p4pde fork
> > if the fix is in the p4pdes fork they make a branch in the p4pdes 
> > fork, which they already have since they used --download-p4pdes and thus 
> >   have the fork on their system
> > they put the fix in new branch in the p4pdes fork and push it
> > they edit p4pdes.py and put a new commit in it pointing to 
> > their branch in the fork
> >run the pipeline again
> >if fails with p4pdes they do the above again
> >else the PETSc branch gets accepted and merged to master
> >   depending on Ed's choice we make an MR for p4pdes depending on 
> > the agreed upon cycle. If Ed puts the fix into his master then we just 
> > update
> >our fork with his latest master with a simple merge of his.  
> > This will then become the new one we test against. If Ed doesn't respond to 
> > the MR all is fine we 
> >   just continue on our fork. If he puts other things in his 
> > branch but not our MR we just merge that into our fork and so are still 
> > testing with his latest master.
> 
> As you've enumerated here, this requires three MRs per change:
> 
>   1. in PETSc with the actual change and to point at a p4pdes commit
>   2. in p4pdes (Ed's or our fork) to implement needed changes (note: this 
> can't merge until #1 

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Jed Brown]

Barry Smith  writes:

>>> git subtree does require use of new commands every time you mess with it 
>>> (say every three months) that we do not know and since each of 
>>> us will do this infrequently it is likely we will not remember them (I 
>>> won't)  while my approach does not require remembering new commands. 
>>> My approach only requires branching, push, and pulling on the fork and 
>>> updating EdsRepo.py which is things all the developers know about and 
>>> do regularly since we update other external packages.
>
>>> Circular dependencies (PETSc CI depends on Ed's p4pdes depends on PETSc) 
>>> are significantly more labor-intensive and it would need to be done on each 
>>> change, versus once per release cycle.
>
> I think we are not communicating on exactly the  same wave length. 
>
> I am not advocating we test on Ed's I am advocating we test on our fork 
> of Ed's and how often our fork gets passed to Ed with MR is a choice all of 
> us make together, now we could do it once at release time with all the fixes 
> we made, every month or each time. Totally up to Ed how often he wants to get 
> them, and he is free to ignore them for weeks (up to the next release if he 
> likes) since our testing will continue because we use our fork. How often we 
> pass them on to Ed is not related to how we maintain our CI.
>
> There is no circular dependency on my approach with Ed's p4pdes.

The circular dependency is on our fork of p4pdes.

> I vote to fix things in our fork or gittree thing continuously since it 
> makes it easier to fix things rather than wait to the release when we try to 
> find and fix everything and it also helps tell us if we introduced a real bug 
> into PETSc and fix PETSc immediately instead of waiting up to 6 months, just 
> like we now we test immediately with Petsc4py and we should do with SLEPc.  
> How often we give the updates to Ed is a completely different issue. 
>
>So again back to my original statement ,it comes down to if the subtree or 
> the fork approach is easier for all the PETSc developers who do not currently 
> know gittree and would need to learn it with your approach. I don't know 
> which is easier learning to use gittree which has its own gotcha's or using 
> mine which we all know but may require an extra step (not involving Ed, just 
> updating the p4pdes.py commit each time we change something in the fork.)
>
>   I think using --download-p4pdes on a couple of systems in the CI is enough, 
> I don't think we need to put it all CI pipelines (I would like slepc in all 
> pipelines).but we could put in all pipelines if we want.
>
>   For completeness I show the exact the work flow for my suggestion
>
>pipelines --download-p4pdes and runs its tests
>if it breaks the developer uses --download-p4pdes  on their system 
>  they fix the problem either by fixing PETSc or what is downloaded from 
> the p4pde fork
> if the fix is in the p4pdes fork they make a branch in the p4pdes 
> fork, which they already have since they used --download-p4pdes and thus 
>   have the fork on their system
> they put the fix in new branch in the p4pdes fork and push it
> they edit p4pdes.py and put a new commit in it pointing to their 
> branch in the fork
>run the pipeline again
>if fails with p4pdes they do the above again
>else the PETSc branch gets accepted and merged to master
>   depending on Ed's choice we make an MR for p4pdes depending on the 
> agreed upon cycle. If Ed puts the fix into his master then we just update
>our fork with his latest master with a simple merge of his.  
> This will then become the new one we test against. If Ed doesn't respond to 
> the MR all is fine we 
>   just continue on our fork. If he puts other things in his 
> branch but not our MR we just merge that into our fork and so are still 
> testing with his latest master.

As you've enumerated here, this requires three MRs per change:

  1. in PETSc with the actual change and to point at a p4pdes commit
  2. in p4pdes (Ed's or our fork) to implement needed changes (note: this can't 
merge until #1 merges)
  3. in PETSc to point at the merge commit of p4pdes (after #2 merges)

With subtree, there is only one MR and it's in the PETSc repository.

Recall that we exchanged hundreds of emails to subtree in BuildSystem long ago, 
then quite a lot more to subtree in petsc4py recently, and now we're doing it 
again.

The arguments against subtreeing p4pdes are

  1. We want an extra hurdle so developers think twice before changing those 
interfaces
  2. We want to drive traffic to Ed's repo
  3. The repository is so big we don't want every `git clone 
gitlab:petsc/petsc` to include it

This repo is small so I'm most sympathetic to #2, but nobody has made these 
arguments yet.

>   This happens to be the exact same thing I do now with slepc and any other 
> git

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Barry Smith]

> On Oct 31, 2020, at 10:59 PM, Jed Brown  wrote:
> 
> Barry Smith  writes:
> 
>>> On Oct 31, 2020, at 5:21 PM, Jed Brown  wrote:
>>> 
>>> Barry Smith  writes:
>>> 
> On Oct 31, 2020, at 9:35 AM, Jed Brown  wrote:
> 
> Barry Smith  writes:
> 
>> I have no problem updating any of Ed's examples if we need to with each 
>> release, so the burdern doesn't fall on him. We simply make a fork of 
>> his 
>> repository with a new branch and update that and make a MR to Ed for 
>> each release and he can have a new branch or tag of his examples for 
>> each new 
>> PETSc release.
>> 
>> Barry
>> 
>> We just make this part of our release process.
> 
> If we add it to CI, the workflow is
> 
 
 I would keep the  fork always petsc/pkg-eds-repo   and have PETSc master 
 always point to the fork. At releases we would have release point directly 
 to eds. 
 This would remove the need to fork constantly I use this model partially 
 for adol-c
 
 Anything wrong with this model?
>>> 
>>> This takes Ed out of the loop, but doesn't avoid the CI circus that I 
>>> outlined. It's more reproducible to always pin to a commit hash in the 
>>> repository, in which case there may be no need to circle back and update 
>>> PETSc after merging in Ed's repo (or our fork).
>>> 
>>> An alternative would be to subtree the examples into PETSc and export upon 
>>> release. Ed could be the CODEOWNER so he can weigh in on proposed changes 
>>> or more idiomatic interfaces that may become available. His repo would 
>>> always work with the latest release.
>> 
>>  What do you mean by release (six month release or all the time with master?)
>> 
>>  I have no problem with alternative approaches but they must truly eliminate 
>> part of the extra work required by my approach on everyone's part not 
>> increase the work. I did find some possible issues with subtree which may or 
>> may not apply 
>> 
>> https://www.atlassian.com/git/tutorials/git-subtree
>> 
>> Drawbacks (but in our opinion they're largely acceptable):
>> 
>> You must learn about a new merge strategy (i.e.git subtree).
>> Contributing code back upstream for the sub-projects is slightly more 
>> complicated.
>> The responsibility of not mixing super and sub-project code in commits lies 
>> with you.
>> 1 and 3 seem a bit concerning, I will certainly make this mistake constantly 
>> without an automatic system to prevent it.
>> 
>> 
>> This is not a good sign, git subtree exists,  but
>> $ git help subtree
>> No manual entry for git-subtree
> 
> You're just using an old Git, for which you need to enable it explicitly.
> 
>> -
>> 
>> Anyways back to my concern.
>> 
>> git subtree does require use of new commands every time you mess with it 
>> (say every three months) that we do not know and since each of 
>> us will do this infrequently it is likely we will not remember them (I 
>> won't)  while my approach does not require remembering new commands. 
>> My approach only requires branching, push, and pulling on the fork and 
>> updating EdsRepo.py which is things all the developers know about and 
>> do regularly since we update other external packages.

>> Circular dependencies (PETSc CI depends on Ed's p4pdes depends on PETSc) are 
>> significantly more labor-intensive and it would need to be done on each 
>> change, versus once per release cycle.

I think we are not communicating on exactly the  same wave length. 

I am not advocating we test on Ed's I am advocating we test on our fork of 
Ed's and how often our fork gets passed to Ed with MR is a choice all of us 
make together, now we could do it once at release time with all the fixes we 
made, every month or each time. Totally up to Ed how often he wants to get 
them, and he is free to ignore them for weeks (up to the next release if he 
likes) since our testing will continue because we use our fork. How often we 
pass them on to Ed is not related to how we maintain our CI.

There is no circular dependency on my approach with Ed's p4pdes.

I vote to fix things in our fork or gittree thing continuously since it 
makes it easier to fix things rather than wait to the release when we try to 
find and fix everything and it also helps tell us if we introduced a real bug 
into PETSc and fix PETSc immediately instead of waiting up to 6 months, just 
like we now we test immediately with Petsc4py and we should do with SLEPc.  How 
often we give the updates to Ed is a completely different issue. 

   So again back to my original statement ,it comes down to if the subtree or 
the fork approach is easier for all the PETSc developers who do not currently 
know gittree and would need to learn it with your approach. I don't know which 
is easier learning to use gittree which has its own gotcha's or using mine 
which we all know but may require an extra step (not involving Ed, just 
updating the p4pdes.py commit each 

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Jed Brown]

Barry Smith  writes:

>> On Oct 31, 2020, at 5:21 PM, Jed Brown  wrote:
>> 
>> Barry Smith  writes:
>> 
 On Oct 31, 2020, at 9:35 AM, Jed Brown  wrote:
 
 Barry Smith  writes:
 
>  I have no problem updating any of Ed's examples if we need to with each 
> release, so the burdern doesn't fall on him. We simply make a fork of his 
> repository with a new branch and update that and make a MR to Ed for each 
> release and he can have a new branch or tag of his examples for each new 
> PETSc release.
> 
>  Barry
> 
> We just make this part of our release process.
 
 If we add it to CI, the workflow is
 
>>> 
>>> I would keep the  fork always petsc/pkg-eds-repo   and have PETSc master 
>>> always point to the fork. At releases we would have release point directly 
>>> to eds. 
>>> This would remove the need to fork constantly I use this model partially 
>>> for adol-c
>>> 
>>>  Anything wrong with this model?
>> 
>> This takes Ed out of the loop, but doesn't avoid the CI circus that I 
>> outlined. It's more reproducible to always pin to a commit hash in the 
>> repository, in which case there may be no need to circle back and update 
>> PETSc after merging in Ed's repo (or our fork).
>> 
>> An alternative would be to subtree the examples into PETSc and export upon 
>> release. Ed could be the CODEOWNER so he can weigh in on proposed changes or 
>> more idiomatic interfaces that may become available. His repo would always 
>> work with the latest release.
>
>   What do you mean by release (six month release or all the time with master?)
>
>   I have no problem with alternative approaches but they must truly eliminate 
> part of the extra work required by my approach on everyone's part not 
> increase the work. I did find some possible issues with subtree which may or 
> may not apply 
>
> https://www.atlassian.com/git/tutorials/git-subtree
>
> Drawbacks (but in our opinion they're largely acceptable):
>
> You must learn about a new merge strategy (i.e.git subtree).
> Contributing code back upstream for the sub-projects is slightly more 
> complicated.
> The responsibility of not mixing super and sub-project code in commits lies 
> with you.
> 1 and 3 seem a bit concerning, I will certainly make this mistake constantly 
> without an automatic system to prevent it.
>
> 
> This is not a good sign, git subtree exists,  but
> $ git help subtree
> No manual entry for git-subtree

You're just using an old Git, for which you need to enable it explicitly.

> -
>
> Anyways back to my concern.
>
> git subtree does require use of new commands every time you mess with it (say 
> every three months) that we do not know and since each of 
> us will do this infrequently it is likely we will not remember them (I won't) 
>  while my approach does not require remembering new commands. 
> My approach only requires branching, push, and pulling on the fork and 
> updating EdsRepo.py which is things all the developers know about and 
> do regularly since we update other external packages.

Circular dependencies (PETSc CI depends on Ed's p4pdes depends on PETSc) are 
significantly more labor-intensive and it would need to be done on each change, 
versus once per release cycle.

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Satish Balay via petsc-dev]

On Sat, 31 Oct 2020, Ed Bueler wrote:

> Maybe I should chime in with what I am thinking.  I'm not thinking dev.
> 
> barry>  I have no problem updating any of Ed's examples if we need to with
> each release, so the burden doesn't fall on him.
> 
> It really isn't a burden to keep the small number of small codes in my repo
> up to date.  I have been doing it for a couple of years now.  The size of
> this "burden" scales with the number of my codes times the rapidity of
> petsc releases; I don't see why this should go out of control.
> 
> On the other hand, keeping my codes up to date _is_ part of authoring the
> book.  While these examples should be kept working with the latest petsc
> release, which forces certain changes, my goal will also be to keep them
> from drifting away from the text of the book.  That is, their documentation
> is essentially fixed, so their feature/API/idiom drift should be
> minimized.  E.g. I will not add functionality to the codes, even if that is
> easy to do, if it makes understanding the originally-intended functionality
> harder, as long as there is a non-deprecated idiom to keep.  I'll take
> seriously all MRs (and issues) on my repo, but I may well not include the
> latest idiomatic petsc.
> 
> If there is ever a second edition that would be the time to bring
> everything fully up to date, syncing (a subset of) the latest petsc api
> with the examples and the text of the book.
> 
> So I guess my question about including my examples into the petsc CI is to
> ask if that is really what you want?

I guess the question is - what issues does CI address. (and the cost). And what 
is the alternative (and cost)

One issue is: updating the examples when interfaces change.

This can be done incrementally [as PETSc changes] and this is CI
friendly. Alternative is to make the changes at release time - in a
single go. This might be more time optimal for you [as from the above
- you have specific goals wrt what changes are ok - and what are not -
so best for petsc developers not to do these changes. so perhaps CI is
not suitable for this issue]

The other issue is: if there is a change in petsc that fundamentally
breaks the examples. And if so - know about this change immediately -
before the change is merged - and re-evaluate the change. CI is
suitable for this issue.

But this would require constant monitoring of all PETSc merge requests
[that break the examples].


>  If you find that one of my codes is
> useful as a regression test then perhaps bring it over and let it evolve
> separately?  Or just set up CI to let me know that there is a fail in my
> latest commit (i.e. the one that you cloned during CI), but don't make the
> whole run fail because of it, and also don't expect to merge my fixes into
> your fork?

The current CI is for feature branches - before merge to master [or
release branch - for fixes].  We can mark a test as 'ignore failure'
[so that the MR is not blocked on this failure].

There will be some coordination issues - as CI is on a feature branch
(so that the feature issues can be addressed before merge to
master). However your examples repo is likely to be in sync with
petsc/master branch (i.e after the feature is merged in). And the
complexity here is to locally test fixes for these issues [i.e
maintain a fork] - or not [let the CI have non-blocking failure until
the issue is resolved in your repo]

Also - you would need different branches for the examples repo so that they
are maintained for each of the petsc releases.

Satish

> 
> Ed
> 
> 
> 
> On Sat, Oct 31, 2020 at 2:21 PM Jed Brown  wrote:
> 
> > Barry Smith  writes:
> >
> > >> On Oct 31, 2020, at 9:35 AM, Jed Brown  wrote:
> > >>
> > >> Barry Smith  writes:
> > >>
> > >>>   I have no problem updating any of Ed's examples if we need to with
> > each release, so the burdern doesn't fall on him. We simply make a fork of
> > his
> > >>> repository with a new branch and update that and make a MR to Ed for
> > each release and he can have a new branch or tag of his examples for each
> > new
> > >>> PETSc release.
> > >>>
> > >>>   Barry
> > >>>
> > >>>  We just make this part of our release process.
> > >>
> > >> If we add it to CI, the workflow is
> > >>
> > >
> > > I would keep the  fork always petsc/pkg-eds-repo   and have PETSc master
> > always point to the fork. At releases we would have release point directly
> > to eds.
> > >  This would remove the need to fork constantly I use this model
> > partially for adol-c
> > >
> > >   Anything wrong with this model?
> >
> > This takes Ed out of the loop, but doesn't avoid the CI circus that I
> > outlined. It's more reproducible to always pin to a commit hash in the
> > repository, in which case there may be no need to circle back and update
> > PETSc after merging in Ed's repo (or our fork).
> >
> > An alternative would be to subtree the examples into PETSc and export upon
> > release. Ed could be the CODEOWNER so he can weigh in on proposed changes
> > o

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Barry Smith]

> On Oct 31, 2020, at 5:55 PM, Ed Bueler  wrote:
> 
> Maybe I should chime in with what I am thinking.  I'm not thinking dev.
> 
> barry>  I have no problem updating any of Ed's examples if we need to with 
> each release, so the burden doesn't fall on him.
> 
> It really isn't a burden to keep the small number of small codes in my repo 
> up to date.  I have been doing it for a couple of years now.  The size of 
> this "burden" scales with the number of my codes times the rapidity of petsc 
> releases; I don't see why this should go out of control.
> 
> On the other hand, keeping my codes up to date _is_ part of authoring the 
> book.  While these examples should be kept working with the latest petsc 
> release, which forces certain changes, my goal will also be to keep them from 
> drifting away from the text of the book.  That is, their documentation is 
> essentially fixed, so their feature/API/idiom drift should be minimized.  
> E.g. I will not add functionality to the codes, even if that is easy to do, 
> if it makes understanding the originally-intended functionality harder, as 
> long as there is a non-deprecated idiom to keep.  I'll take seriously all MRs 
> (and issues) on my repo, but I may well not include the latest idiomatic 
> petsc.
> 
> If there is ever a second edition that would be the time to bring everything 
> fully up to date, syncing (a subset of) the latest petsc api with the 
> examples and the text of the book.
> 
> So I guess my question about including my examples into the petsc CI is to 
> ask if that is really what you want?  If you find that one of my codes is 
> useful as a regression test then perhaps bring it over and let it evolve 
> separately?  Or just set up CI to let me know that there is a fail in my 
> latest commit (i.e. the one that you cloned during CI), but don't make the 
> whole run fail because of it, and also don't expect to merge my fixes into 
> your fork?
> 

  Any fixes you make would get pulled by us into our fork, you would never need 
to do this. Any fixes we make, due to failures in the CI when we change 
something in PETSc that breaks something, would only get into your copy through 
an MR. 

  If we made an idiomatic change in PETSc, we would not make a change to the 
fork to the new idiom, we would insure that our backward compatibility would 
allow your code to continue to work; in fact this would be a great way to test 
to make sure we have done our backward compatibility correctly (currently we 
have no tests of that).

  So our MR would be just like any MR that came in to you from one of your 
readers with a correction to something that failed.

  Of course, all this stuff would be the same regardless of whether we use my 
suggested approach or Jed's subtree approach.

  Barry

> Ed
> 
> 
> 
> On Sat, Oct 31, 2020 at 2:21 PM Jed Brown  > wrote:
> Barry Smith mailto:bsm...@petsc.dev>> writes:
> 
> >> On Oct 31, 2020, at 9:35 AM, Jed Brown  >> > wrote:
> >> 
> >> Barry Smith mailto:bsm...@petsc.dev>> writes:
> >> 
> >>>   I have no problem updating any of Ed's examples if we need to with each 
> >>> release, so the burdern doesn't fall on him. We simply make a fork of his 
> >>> repository with a new branch and update that and make a MR to Ed for each 
> >>> release and he can have a new branch or tag of his examples for each new 
> >>> PETSc release.
> >>> 
> >>>   Barry
> >>> 
> >>>  We just make this part of our release process.
> >> 
> >> If we add it to CI, the workflow is
> >> 
> >
> > I would keep the  fork always petsc/pkg-eds-repo   and have PETSc master 
> > always point to the fork. At releases we would have release point directly 
> > to eds. 
> >  This would remove the need to fork constantly I use this model partially 
> > for adol-c
> >
> >   Anything wrong with this model?
> 
> This takes Ed out of the loop, but doesn't avoid the CI circus that I 
> outlined. It's more reproducible to always pin to a commit hash in the 
> repository, in which case there may be no need to circle back and update 
> PETSc after merging in Ed's repo (or our fork).
> 
> An alternative would be to subtree the examples into PETSc and export upon 
> release. Ed could be the CODEOWNER so he can weigh in on proposed changes or 
> more idiomatic interfaces that may become available. His repo would always 
> work with the latest release.
> 
> 
> -- 
> Ed Bueler
> Dept of Mathematics and Statistics
> University of Alaska Fairbanks
> Fairbanks, AK 99775-6660
> 306C Chapman

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Barry Smith]

> On Oct 31, 2020, at 5:21 PM, Jed Brown  wrote:
> 
> Barry Smith  writes:
> 
>>> On Oct 31, 2020, at 9:35 AM, Jed Brown  wrote:
>>> 
>>> Barry Smith  writes:
>>> 
  I have no problem updating any of Ed's examples if we need to with each 
 release, so the burdern doesn't fall on him. We simply make a fork of his 
 repository with a new branch and update that and make a MR to Ed for each 
 release and he can have a new branch or tag of his examples for each new 
 PETSc release.
 
  Barry
 
 We just make this part of our release process.
>>> 
>>> If we add it to CI, the workflow is
>>> 
>> 
>> I would keep the  fork always petsc/pkg-eds-repo   and have PETSc master 
>> always point to the fork. At releases we would have release point directly 
>> to eds. 
>> This would remove the need to fork constantly I use this model partially for 
>> adol-c
>> 
>>  Anything wrong with this model?
> 
> This takes Ed out of the loop, but doesn't avoid the CI circus that I 
> outlined. It's more reproducible to always pin to a commit hash in the 
> repository, in which case there may be no need to circle back and update 
> PETSc after merging in Ed's repo (or our fork).
> 
> An alternative would be to subtree the examples into PETSc and export upon 
> release. Ed could be the CODEOWNER so he can weigh in on proposed changes or 
> more idiomatic interfaces that may become available. His repo would always 
> work with the latest release.

  What do you mean by release (six month release or all the time with master?)

  I have no problem with alternative approaches but they must truly eliminate 
part of the extra work required by my approach on everyone's part not increase 
the work. I did find some possible issues with subtree which may or may not 
apply 

https://www.atlassian.com/git/tutorials/git-subtree

Drawbacks (but in our opinion they're largely acceptable):

You must learn about a new merge strategy (i.e.git subtree).
Contributing code back upstream for the sub-projects is slightly more 
complicated.
The responsibility of not mixing super and sub-project code in commits lies 
with you.
1 and 3 seem a bit concerning, I will certainly make this mistake constantly 
without an automatic system to prevent it.


This is not a good sign, git subtree exists,  but
$ git help subtree
No manual entry for git-subtree

-

Anyways back to my concern.

git subtree does require use of new commands every time you mess with it (say 
every three months) that we do not know and since each of 
us will do this infrequently it is likely we will not remember them (I won't)  
while my approach does not require remembering new commands. 
My approach only requires branching, push, and pulling on the fork and updating 
EdsRepo.py which is things all the developers know about and 
do regularly since we update other external packages.

If we all needed to use git subtree often then the extra burden of learning it 
and remembering it would just happen, but since we are 
likely to use it infrequently it is not clear to me that the extra burden of 
everyone learning subtree is truly simpler than my approach for everyone but 
you. 

Also making Ed a code owner seems to give him an extra unneeded burden by 
sending mail for every trivial fix, that he would not face if we just submitted 
MR to him when anything dramatically changes and at release time.


I would just put the fork of Ed's repository in a couple of pipeline tests with 
--download-edsrepo then people don't need to monkey with it themselves unless 
there is a failure and if there is a failure they proceed just like any other 
external package, add --download-edsrepository locally, fix in the downloaded 
repo, push, change the commit in edsrepo.py and push. 

In conclusion, if you can demonstrate that git subtree is truly simpler for all 
of us including Ed, then definitely we should use it. But I have my concerns 
that it actually adds a good amount of unneeded new complexity. 

 Barry


 

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Ed Bueler]

Maybe I should chime in with what I am thinking.  I'm not thinking dev.

barry>  I have no problem updating any of Ed's examples if we need to with
each release, so the burden doesn't fall on him.

It really isn't a burden to keep the small number of small codes in my repo
up to date.  I have been doing it for a couple of years now.  The size of
this "burden" scales with the number of my codes times the rapidity of
petsc releases; I don't see why this should go out of control.

On the other hand, keeping my codes up to date _is_ part of authoring the
book.  While these examples should be kept working with the latest petsc
release, which forces certain changes, my goal will also be to keep them
from drifting away from the text of the book.  That is, their documentation
is essentially fixed, so their feature/API/idiom drift should be
minimized.  E.g. I will not add functionality to the codes, even if that is
easy to do, if it makes understanding the originally-intended functionality
harder, as long as there is a non-deprecated idiom to keep.  I'll take
seriously all MRs (and issues) on my repo, but I may well not include the
latest idiomatic petsc.

If there is ever a second edition that would be the time to bring
everything fully up to date, syncing (a subset of) the latest petsc api
with the examples and the text of the book.

So I guess my question about including my examples into the petsc CI is to
ask if that is really what you want?  If you find that one of my codes is
useful as a regression test then perhaps bring it over and let it evolve
separately?  Or just set up CI to let me know that there is a fail in my
latest commit (i.e. the one that you cloned during CI), but don't make the
whole run fail because of it, and also don't expect to merge my fixes into
your fork?

Ed



On Sat, Oct 31, 2020 at 2:21 PM Jed Brown  wrote:

> Barry Smith  writes:
>
> >> On Oct 31, 2020, at 9:35 AM, Jed Brown  wrote:
> >>
> >> Barry Smith  writes:
> >>
> >>>   I have no problem updating any of Ed's examples if we need to with
> each release, so the burdern doesn't fall on him. We simply make a fork of
> his
> >>> repository with a new branch and update that and make a MR to Ed for
> each release and he can have a new branch or tag of his examples for each
> new
> >>> PETSc release.
> >>>
> >>>   Barry
> >>>
> >>>  We just make this part of our release process.
> >>
> >> If we add it to CI, the workflow is
> >>
> >
> > I would keep the  fork always petsc/pkg-eds-repo   and have PETSc master
> always point to the fork. At releases we would have release point directly
> to eds.
> >  This would remove the need to fork constantly I use this model
> partially for adol-c
> >
> >   Anything wrong with this model?
>
> This takes Ed out of the loop, but doesn't avoid the CI circus that I
> outlined. It's more reproducible to always pin to a commit hash in the
> repository, in which case there may be no need to circle back and update
> PETSc after merging in Ed's repo (or our fork).
>
> An alternative would be to subtree the examples into PETSc and export upon
> release. Ed could be the CODEOWNER so he can weigh in on proposed changes
> or more idiomatic interfaces that may become available. His repo would
> always work with the latest release.
>


-- 
Ed Bueler
Dept of Mathematics and Statistics
University of Alaska Fairbanks
Fairbanks, AK 99775-6660
306C Chapman

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Jed Brown]

Barry Smith  writes:

>> On Oct 31, 2020, at 9:35 AM, Jed Brown  wrote:
>> 
>> Barry Smith  writes:
>> 
>>>   I have no problem updating any of Ed's examples if we need to with each 
>>> release, so the burdern doesn't fall on him. We simply make a fork of his 
>>> repository with a new branch and update that and make a MR to Ed for each 
>>> release and he can have a new branch or tag of his examples for each new 
>>> PETSc release.
>>> 
>>>   Barry
>>> 
>>>  We just make this part of our release process.
>> 
>> If we add it to CI, the workflow is
>> 
>
> I would keep the  fork always petsc/pkg-eds-repo   and have PETSc master 
> always point to the fork. At releases we would have release point directly to 
> eds. 
>  This would remove the need to fork constantly I use this model partially for 
> adol-c
>
>   Anything wrong with this model?

This takes Ed out of the loop, but doesn't avoid the CI circus that I outlined. 
It's more reproducible to always pin to a commit hash in the repository, in 
which case there may be no need to circle back and update PETSc after merging 
in Ed's repo (or our fork).

An alternative would be to subtree the examples into PETSc and export upon 
release. Ed could be the CODEOWNER so he can weigh in on proposed changes or 
more idiomatic interfaces that may become available. His repo would always work 
with the latest release.

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Barry Smith]

> On Oct 31, 2020, at 9:35 AM, Jed Brown  wrote:
> 
> Barry Smith  writes:
> 
>>   I have no problem updating any of Ed's examples if we need to with each 
>> release, so the burdern doesn't fall on him. We simply make a fork of his 
>> repository with a new branch and update that and make a MR to Ed for each 
>> release and he can have a new branch or tag of his examples for each new 
>> PETSc release.
>> 
>>   Barry
>> 
>>  We just make this part of our release process.
> 
> If we add it to CI, the workflow is
> 

I would keep the  fork always petsc/pkg-eds-repo   and have PETSc master always 
point to the fork. At releases we would have release point directly to eds. 
 This would remove the need to fork constantly I use this model partially for 
adol-c

  Anything wrong with this model?

Barry



> 1. Do work in the PETSc repository
> 2. Fork Ed's repository, create branch, update, and push
> 3. Point PETSc CI to the fork


> 4. Submit PETSc MR, review, and merge
> 5. Submit PR to Ed's repo, review, and merge


> 6. Update PETSc CI to once again point at Ed's "main" branch

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Satish Balay via petsc-dev]

On Sat, 31 Oct 2020, Jed Brown wrote:

> Satish Balay  writes:
> 
> > On Sat, 31 Oct 2020, Jed Brown wrote:
> >
> >> Barry Smith  writes:
> >> 
> >> >I have no problem updating any of Ed's examples if we need to with 
> >> > each release, so the burdern doesn't fall on him. We simply make a fork 
> >> > of his 
> >> > repository with a new branch and update that and make a MR to Ed for 
> >> > each release and he can have a new branch or tag of his examples for 
> >> > each new 
> >> > PETSc release.
> >> >
> >> >Barry
> >> >
> >> >   We just make this part of our release process.
> >> 
> >> If we add it to CI, the workflow is
> >> 
> >> 1. Do work in the PETSc repository
> >> 2. Fork Ed's repository, create branch, update, and push
> >> 3. Point PETSc CI to the fork
> >> 4. Submit PETSc MR, review, and merge
> >> 5. Submit PR to Ed's repo, review, and merge
> >> 6. Update PETSc CI to once again point at Ed's "main" branch
> >
> > we had similar workflow with petsc4py before it was merged into petsc repo.
> 
> petsc4py tests didn't run as part of CI, we just fixed issues when they were 
> reported or a developer encountered them. We merged it into the PETSc 
> repository largely because adding it to CI as an external repo would be too 
> taxing.

We didn't run petsc4py test suite - but we have a minimal petsc4py test. So if 
there was a change in petsc (feature branch - in MR) that broke petsc4py - it 
was found - and fixed [before this MR was merged]


And yes - one difference here would be - we would have to both build and run 
the tests in this repo.

Satish

> 
> > primary difference: a branch in a fork vs branch in upstream petsc4py
> > [alternative to switch back and forth with fork vs main repo is always have 
> > the fork be mirrored..]
> 

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Jed Brown]

Satish Balay  writes:

> On Sat, 31 Oct 2020, Jed Brown wrote:
>
>> Barry Smith  writes:
>> 
>> >I have no problem updating any of Ed's examples if we need to with each 
>> > release, so the burdern doesn't fall on him. We simply make a fork of his 
>> > repository with a new branch and update that and make a MR to Ed for each 
>> > release and he can have a new branch or tag of his examples for each new 
>> > PETSc release.
>> >
>> >Barry
>> >
>> >   We just make this part of our release process.
>> 
>> If we add it to CI, the workflow is
>> 
>> 1. Do work in the PETSc repository
>> 2. Fork Ed's repository, create branch, update, and push
>> 3. Point PETSc CI to the fork
>> 4. Submit PETSc MR, review, and merge
>> 5. Submit PR to Ed's repo, review, and merge
>> 6. Update PETSc CI to once again point at Ed's "main" branch
>
> we had similar workflow with petsc4py before it was merged into petsc repo.

petsc4py tests didn't run as part of CI, we just fixed issues when they were 
reported or a developer encountered them. We merged it into the PETSc 
repository largely because adding it to CI as an external repo would be too 
taxing.

> primary difference: a branch in a fork vs branch in upstream petsc4py
> [alternative to switch back and forth with fork vs main repo is always have 
> the fork be mirrored..]

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Satish Balay via petsc-dev]

On Sat, 31 Oct 2020, Jed Brown wrote:

> Barry Smith  writes:
> 
> >I have no problem updating any of Ed's examples if we need to with each 
> > release, so the burdern doesn't fall on him. We simply make a fork of his 
> > repository with a new branch and update that and make a MR to Ed for each 
> > release and he can have a new branch or tag of his examples for each new 
> > PETSc release.
> >
> >Barry
> >
> >   We just make this part of our release process.
> 
> If we add it to CI, the workflow is
> 
> 1. Do work in the PETSc repository
> 2. Fork Ed's repository, create branch, update, and push
> 3. Point PETSc CI to the fork
> 4. Submit PETSc MR, review, and merge
> 5. Submit PR to Ed's repo, review, and merge
> 6. Update PETSc CI to once again point at Ed's "main" branch

we had similar workflow with petsc4py before it was merged into petsc repo.

primary difference: a branch in a fork vs branch in upstream petsc4py
[alternative to switch back and forth with fork vs main repo is always have the 
fork be mirrored..]

Satish

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Jed Brown]

Barry Smith  writes:

>I have no problem updating any of Ed's examples if we need to with each 
> release, so the burdern doesn't fall on him. We simply make a fork of his 
> repository with a new branch and update that and make a MR to Ed for each 
> release and he can have a new branch or tag of his examples for each new 
> PETSc release.
>
>Barry
>
>   We just make this part of our release process.

If we add it to CI, the workflow is

1. Do work in the PETSc repository
2. Fork Ed's repository, create branch, update, and push
3. Point PETSc CI to the fork
4. Submit PETSc MR, review, and merge
5. Submit PR to Ed's repo, review, and merge
6. Update PETSc CI to once again point at Ed's "main" branch

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Barry Smith]

   I have no problem updating any of Ed's examples if we need to with each 
release, so the burdern doesn't fall on him. We simply make a fork of his 
repository with a new branch and update that and make a MR to Ed for each 
release and he can have a new branch or tag of his examples for each new 
PETSc release.

   Barry

  We just make this part of our release process.

> On Oct 30, 2020, at 9:38 PM, Jed Brown  wrote:
> 
> This is fine until an API for 3.15 changes and we need to sync with Ed's 
> repository to make the tests pass. (You could argue we shouldn't do that to 
> the interfaces that Ed has used for N releases, but it's bound to happen 
> eventually.)
> 
> Junchao Zhang  writes:
> 
>> Ed,
>>  I agree with all what you said.  My thought is we don't need to add each
>> of your examples into corresponding src/XX/tutorials/.  Your repo can be a
>> standalone directory and we just need PETSc CI to be able to run them.
>> --Junchao Zhang
>> 
>> 
>> On Fri, Oct 30, 2020 at 9:00 PM Ed Bueler  wrote:
>> 
>>> Junchao--
>>> 
 I was wondering if it is feasible to add your example programs to PETSc
>>> tests so that readers will always be able to run your code.
>>> 
>>> Thanks for asking.  There was a deliberate idea here, which I want
>>> to explain, and the petsc-dev list is the right spot.  (Sorry if this is
>>> more than you want to know.)
>>> 
>>> First, the example programs are in a completely public spot:
>>>https://github.com/bueler/p4pdes
>>> No one needs to own the book to run the codes, for example.  I welcome
>>> corrections/feedback/improvements through the issues interface at that
>>> repo, whether or not connected they are connected to the book text.
>>> 
>>> In fact, here are things one may observe about the petsc tutorial
>>> src/XX/tutorials/exN.c) examples:
>>> 
>>> 1.  They may or may not point to a clear document(s) which can help a
>>> beginner know how they are designed.
>>> 2.  They don't have a uniform style because of different authorship.
>>> 3.  They are not ordered by difficulty in any clear way.  (E.g. ex1.c may
>>> not be the best example to start with, and the beginner would not be able
>>> to grep to find "easy" even if they can find some function from the API
>>> that way.)
>>> 4.  Their features evolve over time as developers work with the examples
>>> as regression/feature tests.  (See src/snes/tutorials/ex5.c.)
>>> 
>>> Note that all of these facts are exactly what petsc devs would want!  That
>>> is, the way the set of examples in the petsc tree are structured helps with
>>> fast development by a diverse dev team.
>>> 
>>> However the same facts make the examples less friendly to those who don't
>>> already know petsc.  Thus my opinion about the book's example codes is that
>>> a single source of stable examples, ordered by difficulty, closely tied to
>>> beginner documentation, of uniform style, and kinda boring to most petsc
>>> devs, is something I can supply and maintain.  So I'll be acting as editor
>>> to preserve the intent and simplicity of the examples.
>>> 
>>> Does that make sense?
>>> 
>>> Needless to say, fork my repo all you want!  The MIT license is nice and
>>> permissive.
>>> 
>>> Ed
>>> 
>>> 
>>> 
>>> On Fri, Oct 30, 2020 at 4:50 PM Junchao Zhang 
>>> wrote:
>>> 
 Prof. Ed Bueler,
   Congratulations on your book. I am eager to read it.
   I was wondering if it is feasible to add your example programs to
 PETSc tests so that readers will always be able to run your code.
 --Junchao Zhang
 
 
 On Thu, Oct 29, 2020 at 8:29 PM Ed Bueler  wrote:
 
> All --
> 
> SIAM Press just published my new book "PETSc for Partial Differential
> Equations: Numerical Solutions in C and Python":
> 
>  https://my.siam.org/Store/Product/viewproduct/?ProductId=32850137
> 
> The book is available both as a paperback and an e-book with working
> links.  A SIAM member discount is available, of course.
> 
> This book is a genuine introduction which does not assume you have used
> PETSc before, and which should make sense even if your differential
> equations knowledge is basic.  The prerequisites are a bit of programming
> in C and a bit of numerical linear algebra, roughly like the main ideas of
> Trefethen and Bau, but even that is reviewed and summarized.  I've made an
> effort to introduce discretizations from the beginning, especially finite
> differences and elements.
> 
> The book is based on a collection of example programs at
> https://github.com/bueler/p4pdes.  Most of these codes call PETSc
> directly through the C API, but the last two chapters have Python codes
> using UFL and Firedrake.  Nonetheless the book contains ideas, 
> mathematical
> and computational; it complements, but does not replace, the PETSc User's
> Manual and the tutorial examples in the PETSc source.  Concepts are
> explained and illustrated, with s

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Ed Bueler]

Junchao--

> I was wondering if it is feasible to add your example programs to PETSc
tests so that readers will always be able to run your code.

Thanks for asking.  There was a deliberate idea here, which I want
to explain, and the petsc-dev list is the right spot.  (Sorry if this is
more than you want to know.)

First, the example programs are in a completely public spot:
https://github.com/bueler/p4pdes
No one needs to own the book to run the codes, for example.  I welcome
corrections/feedback/improvements through the issues interface at that
repo, whether or not connected they are connected to the book text.

In fact, here are things one may observe about the petsc tutorial
src/XX/tutorials/exN.c) examples:

1.  They may or may not point to a clear document(s) which can help a
beginner know how they are designed.
2.  They don't have a uniform style because of different authorship.
3.  They are not ordered by difficulty in any clear way.  (E.g. ex1.c may
not be the best example to start with, and the beginner would not be able
to grep to find "easy" even if they can find some function from the API
that way.)
4.  Their features evolve over time as developers work with the examples as
regression/feature tests.  (See src/snes/tutorials/ex5.c.)

Note that all of these facts are exactly what petsc devs would want!  That
is, the way the set of examples in the petsc tree are structured helps with
fast development by a diverse dev team.

However the same facts make the examples less friendly to those who don't
already know petsc.  Thus my opinion about the book's example codes is that
a single source of stable examples, ordered by difficulty, closely tied to
beginner documentation, of uniform style, and kinda boring to most petsc
devs, is something I can supply and maintain.  So I'll be acting as editor
to preserve the intent and simplicity of the examples.

Does that make sense?

Needless to say, fork my repo all you want!  The MIT license is nice and
permissive.

Ed



On Fri, Oct 30, 2020 at 4:50 PM Junchao Zhang 
wrote:

> Prof. Ed Bueler,
>Congratulations on your book. I am eager to read it.
>I was wondering if it is feasible to add your example programs to PETSc
> tests so that readers will always be able to run your code.
> --Junchao Zhang
>
>
> On Thu, Oct 29, 2020 at 8:29 PM Ed Bueler  wrote:
>
>> All --
>>
>> SIAM Press just published my new book "PETSc for Partial Differential
>> Equations: Numerical Solutions in C and Python":
>>
>>   https://my.siam.org/Store/Product/viewproduct/?ProductId=32850137
>>
>> The book is available both as a paperback and an e-book with working
>> links.  A SIAM member discount is available, of course.
>>
>> This book is a genuine introduction which does not assume you have used
>> PETSc before, and which should make sense even if your differential
>> equations knowledge is basic.  The prerequisites are a bit of programming
>> in C and a bit of numerical linear algebra, roughly like the main ideas of
>> Trefethen and Bau, but even that is reviewed and summarized.  I've made an
>> effort to introduce discretizations from the beginning, especially finite
>> differences and elements.
>>
>> The book is based on a collection of example programs at
>> https://github.com/bueler/p4pdes.  Most of these codes call PETSc
>> directly through the C API, but the last two chapters have Python codes
>> using UFL and Firedrake.  Nonetheless the book contains ideas, mathematical
>> and computational; it complements, but does not replace, the PETSc User's
>> Manual and the tutorial examples in the PETSc source.  Concepts are
>> explained and illustrated, with sufficient context to facilitate further
>> development. Performance (optimality) and parallel scalability are the
>> primary goals, so preconditioners including multigrid are central threads,
>> and run-time solver options are explored in both the text and the exercises.
>>
>> Here is the place to appreciate the usual PETSc suspects for their
>> comments on drafts, and help in writing this book: Barry, Jed, Matt, Dave,
>> Rich, Lois, Patrick, Mark, Satish, David K., and many others.  Also let me
>> say that SIAM Press has nothing but professionals who are nice to work with
>> too; send them your book idea!
>>
>> Ed
>>
>> --
>> Ed Bueler
>> Dept of Mathematics and Statistics
>> University of Alaska Fairbanks
>> Fairbanks, AK 99775-6660
>> 306C Chapman
>>
>

-- 
Ed Bueler
Dept of Mathematics and Statistics
University of Alaska Fairbanks
Fairbanks, AK 99775-6660
306C Chapman

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Ed Bueler]

I'm no expert on CI, much less PETSc's.  But is there any reason not to
clone my repo as the first CI step to do what you say?

Ed

On Fri, Oct 30, 2020 at 6:11 PM Junchao Zhang 
wrote:

> Ed,
>   I agree with all what you said.  My thought is we don't need to add each
> of your examples into corresponding src/XX/tutorials/.  Your repo can be a
> standalone directory and we just need PETSc CI to be able to run them.
> --Junchao Zhang
>
>
> On Fri, Oct 30, 2020 at 9:00 PM Ed Bueler  wrote:
>
>> Junchao--
>>
>> > I was wondering if it is feasible to add your example programs to PETSc
>> tests so that readers will always be able to run your code.
>>
>> Thanks for asking.  There was a deliberate idea here, which I want
>> to explain, and the petsc-dev list is the right spot.  (Sorry if this is
>> more than you want to know.)
>>
>> First, the example programs are in a completely public spot:
>> https://github.com/bueler/p4pdes
>> No one needs to own the book to run the codes, for example.  I welcome
>> corrections/feedback/improvements through the issues interface at that
>> repo, whether or not connected they are connected to the book text.
>>
>> In fact, here are things one may observe about the petsc tutorial
>> src/XX/tutorials/exN.c) examples:
>>
>> 1.  They may or may not point to a clear document(s) which can help a
>> beginner know how they are designed.
>> 2.  They don't have a uniform style because of different authorship.
>> 3.  They are not ordered by difficulty in any clear way.  (E.g. ex1.c may
>> not be the best example to start with, and the beginner would not be able
>> to grep to find "easy" even if they can find some function from the API
>> that way.)
>> 4.  Their features evolve over time as developers work with the examples
>> as regression/feature tests.  (See src/snes/tutorials/ex5.c.)
>>
>> Note that all of these facts are exactly what petsc devs would want!
>> That is, the way the set of examples in the petsc tree are structured helps
>> with fast development by a diverse dev team.
>>
>> However the same facts make the examples less friendly to those who don't
>> already know petsc.  Thus my opinion about the book's example codes is that
>> a single source of stable examples, ordered by difficulty, closely tied to
>> beginner documentation, of uniform style, and kinda boring to most petsc
>> devs, is something I can supply and maintain.  So I'll be acting as editor
>> to preserve the intent and simplicity of the examples.
>>
>> Does that make sense?
>>
>> Needless to say, fork my repo all you want!  The MIT license is nice and
>> permissive.
>>
>> Ed
>>
>>
>>
>> On Fri, Oct 30, 2020 at 4:50 PM Junchao Zhang 
>> wrote:
>>
>>> Prof. Ed Bueler,
>>>Congratulations on your book. I am eager to read it.
>>>I was wondering if it is feasible to add your example programs to
>>> PETSc tests so that readers will always be able to run your code.
>>> --Junchao Zhang
>>>
>>>
>>> On Thu, Oct 29, 2020 at 8:29 PM Ed Bueler  wrote:
>>>
 All --

 SIAM Press just published my new book "PETSc for Partial Differential
 Equations: Numerical Solutions in C and Python":

   https://my.siam.org/Store/Product/viewproduct/?ProductId=32850137

 The book is available both as a paperback and an e-book with working
 links.  A SIAM member discount is available, of course.

 This book is a genuine introduction which does not assume you have used
 PETSc before, and which should make sense even if your differential
 equations knowledge is basic.  The prerequisites are a bit of programming
 in C and a bit of numerical linear algebra, roughly like the main ideas of
 Trefethen and Bau, but even that is reviewed and summarized.  I've made an
 effort to introduce discretizations from the beginning, especially finite
 differences and elements.

 The book is based on a collection of example programs at
 https://github.com/bueler/p4pdes.  Most of these codes call PETSc
 directly through the C API, but the last two chapters have Python codes
 using UFL and Firedrake.  Nonetheless the book contains ideas, mathematical
 and computational; it complements, but does not replace, the PETSc User's
 Manual and the tutorial examples in the PETSc source.  Concepts are
 explained and illustrated, with sufficient context to facilitate further
 development. Performance (optimality) and parallel scalability are the
 primary goals, so preconditioners including multigrid are central threads,
 and run-time solver options are explored in both the text and the 
 exercises.

 Here is the place to appreciate the usual PETSc suspects for their
 comments on drafts, and help in writing this book: Barry, Jed, Matt, Dave,
 Rich, Lois, Patrick, Mark, Satish, David K., and many others.  Also let me
 say that SIAM Press has nothing but professionals who are nice to work with
 too; send them your book idea!
>>>

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Jed Brown]

This is fine until an API for 3.15 changes and we need to sync with Ed's 
repository to make the tests pass. (You could argue we shouldn't do that to the 
interfaces that Ed has used for N releases, but it's bound to happen 
eventually.)

Junchao Zhang  writes:

> Ed,
>   I agree with all what you said.  My thought is we don't need to add each
> of your examples into corresponding src/XX/tutorials/.  Your repo can be a
> standalone directory and we just need PETSc CI to be able to run them.
> --Junchao Zhang
>
>
> On Fri, Oct 30, 2020 at 9:00 PM Ed Bueler  wrote:
>
>> Junchao--
>>
>> > I was wondering if it is feasible to add your example programs to PETSc
>> tests so that readers will always be able to run your code.
>>
>> Thanks for asking.  There was a deliberate idea here, which I want
>> to explain, and the petsc-dev list is the right spot.  (Sorry if this is
>> more than you want to know.)
>>
>> First, the example programs are in a completely public spot:
>> https://github.com/bueler/p4pdes
>> No one needs to own the book to run the codes, for example.  I welcome
>> corrections/feedback/improvements through the issues interface at that
>> repo, whether or not connected they are connected to the book text.
>>
>> In fact, here are things one may observe about the petsc tutorial
>> src/XX/tutorials/exN.c) examples:
>>
>> 1.  They may or may not point to a clear document(s) which can help a
>> beginner know how they are designed.
>> 2.  They don't have a uniform style because of different authorship.
>> 3.  They are not ordered by difficulty in any clear way.  (E.g. ex1.c may
>> not be the best example to start with, and the beginner would not be able
>> to grep to find "easy" even if they can find some function from the API
>> that way.)
>> 4.  Their features evolve over time as developers work with the examples
>> as regression/feature tests.  (See src/snes/tutorials/ex5.c.)
>>
>> Note that all of these facts are exactly what petsc devs would want!  That
>> is, the way the set of examples in the petsc tree are structured helps with
>> fast development by a diverse dev team.
>>
>> However the same facts make the examples less friendly to those who don't
>> already know petsc.  Thus my opinion about the book's example codes is that
>> a single source of stable examples, ordered by difficulty, closely tied to
>> beginner documentation, of uniform style, and kinda boring to most petsc
>> devs, is something I can supply and maintain.  So I'll be acting as editor
>> to preserve the intent and simplicity of the examples.
>>
>> Does that make sense?
>>
>> Needless to say, fork my repo all you want!  The MIT license is nice and
>> permissive.
>>
>> Ed
>>
>>
>>
>> On Fri, Oct 30, 2020 at 4:50 PM Junchao Zhang 
>> wrote:
>>
>>> Prof. Ed Bueler,
>>>Congratulations on your book. I am eager to read it.
>>>I was wondering if it is feasible to add your example programs to
>>> PETSc tests so that readers will always be able to run your code.
>>> --Junchao Zhang
>>>
>>>
>>> On Thu, Oct 29, 2020 at 8:29 PM Ed Bueler  wrote:
>>>
 All --

 SIAM Press just published my new book "PETSc for Partial Differential
 Equations: Numerical Solutions in C and Python":

   https://my.siam.org/Store/Product/viewproduct/?ProductId=32850137

 The book is available both as a paperback and an e-book with working
 links.  A SIAM member discount is available, of course.

 This book is a genuine introduction which does not assume you have used
 PETSc before, and which should make sense even if your differential
 equations knowledge is basic.  The prerequisites are a bit of programming
 in C and a bit of numerical linear algebra, roughly like the main ideas of
 Trefethen and Bau, but even that is reviewed and summarized.  I've made an
 effort to introduce discretizations from the beginning, especially finite
 differences and elements.

 The book is based on a collection of example programs at
 https://github.com/bueler/p4pdes.  Most of these codes call PETSc
 directly through the C API, but the last two chapters have Python codes
 using UFL and Firedrake.  Nonetheless the book contains ideas, mathematical
 and computational; it complements, but does not replace, the PETSc User's
 Manual and the tutorial examples in the PETSc source.  Concepts are
 explained and illustrated, with sufficient context to facilitate further
 development. Performance (optimality) and parallel scalability are the
 primary goals, so preconditioners including multigrid are central threads,
 and run-time solver options are explored in both the text and the 
 exercises.

 Here is the place to appreciate the usual PETSc suspects for their
 comments on drafts, and help in writing this book: Barry, Jed, Matt, Dave,
 Rich, Lois, Patrick, Mark, Satish, David K., and many others.  Also let me
 say that SIAM Press has nothing but pr

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Junchao Zhang]

Ed,
  I agree with all what you said.  My thought is we don't need to add each
of your examples into corresponding src/XX/tutorials/.  Your repo can be a
standalone directory and we just need PETSc CI to be able to run them.
--Junchao Zhang


On Fri, Oct 30, 2020 at 9:00 PM Ed Bueler  wrote:

> Junchao--
>
> > I was wondering if it is feasible to add your example programs to PETSc
> tests so that readers will always be able to run your code.
>
> Thanks for asking.  There was a deliberate idea here, which I want
> to explain, and the petsc-dev list is the right spot.  (Sorry if this is
> more than you want to know.)
>
> First, the example programs are in a completely public spot:
> https://github.com/bueler/p4pdes
> No one needs to own the book to run the codes, for example.  I welcome
> corrections/feedback/improvements through the issues interface at that
> repo, whether or not connected they are connected to the book text.
>
> In fact, here are things one may observe about the petsc tutorial
> src/XX/tutorials/exN.c) examples:
>
> 1.  They may or may not point to a clear document(s) which can help a
> beginner know how they are designed.
> 2.  They don't have a uniform style because of different authorship.
> 3.  They are not ordered by difficulty in any clear way.  (E.g. ex1.c may
> not be the best example to start with, and the beginner would not be able
> to grep to find "easy" even if they can find some function from the API
> that way.)
> 4.  Their features evolve over time as developers work with the examples
> as regression/feature tests.  (See src/snes/tutorials/ex5.c.)
>
> Note that all of these facts are exactly what petsc devs would want!  That
> is, the way the set of examples in the petsc tree are structured helps with
> fast development by a diverse dev team.
>
> However the same facts make the examples less friendly to those who don't
> already know petsc.  Thus my opinion about the book's example codes is that
> a single source of stable examples, ordered by difficulty, closely tied to
> beginner documentation, of uniform style, and kinda boring to most petsc
> devs, is something I can supply and maintain.  So I'll be acting as editor
> to preserve the intent and simplicity of the examples.
>
> Does that make sense?
>
> Needless to say, fork my repo all you want!  The MIT license is nice and
> permissive.
>
> Ed
>
>
>
> On Fri, Oct 30, 2020 at 4:50 PM Junchao Zhang 
> wrote:
>
>> Prof. Ed Bueler,
>>Congratulations on your book. I am eager to read it.
>>I was wondering if it is feasible to add your example programs to
>> PETSc tests so that readers will always be able to run your code.
>> --Junchao Zhang
>>
>>
>> On Thu, Oct 29, 2020 at 8:29 PM Ed Bueler  wrote:
>>
>>> All --
>>>
>>> SIAM Press just published my new book "PETSc for Partial Differential
>>> Equations: Numerical Solutions in C and Python":
>>>
>>>   https://my.siam.org/Store/Product/viewproduct/?ProductId=32850137
>>>
>>> The book is available both as a paperback and an e-book with working
>>> links.  A SIAM member discount is available, of course.
>>>
>>> This book is a genuine introduction which does not assume you have used
>>> PETSc before, and which should make sense even if your differential
>>> equations knowledge is basic.  The prerequisites are a bit of programming
>>> in C and a bit of numerical linear algebra, roughly like the main ideas of
>>> Trefethen and Bau, but even that is reviewed and summarized.  I've made an
>>> effort to introduce discretizations from the beginning, especially finite
>>> differences and elements.
>>>
>>> The book is based on a collection of example programs at
>>> https://github.com/bueler/p4pdes.  Most of these codes call PETSc
>>> directly through the C API, but the last two chapters have Python codes
>>> using UFL and Firedrake.  Nonetheless the book contains ideas, mathematical
>>> and computational; it complements, but does not replace, the PETSc User's
>>> Manual and the tutorial examples in the PETSc source.  Concepts are
>>> explained and illustrated, with sufficient context to facilitate further
>>> development. Performance (optimality) and parallel scalability are the
>>> primary goals, so preconditioners including multigrid are central threads,
>>> and run-time solver options are explored in both the text and the exercises.
>>>
>>> Here is the place to appreciate the usual PETSc suspects for their
>>> comments on drafts, and help in writing this book: Barry, Jed, Matt, Dave,
>>> Rich, Lois, Patrick, Mark, Satish, David K., and many others.  Also let me
>>> say that SIAM Press has nothing but professionals who are nice to work with
>>> too; send them your book idea!
>>>
>>> Ed
>>>
>>> --
>>> Ed Bueler
>>> Dept of Mathematics and Statistics
>>> University of Alaska Fairbanks
>>> Fairbanks, AK 99775-6660
>>> 306C Chapman
>>>
>>
>
> --
> Ed Bueler
> Dept of Mathematics and Statistics
> University of Alaska Fairbanks
> Fairbanks, AK 99775-6660
> 306C Chapman
>

Re: [petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Junchao Zhang]

Prof. Ed Bueler,
   Congratulations on your book. I am eager to read it.
   I was wondering if it is feasible to add your example programs to PETSc
tests so that readers will always be able to run your code.
--Junchao Zhang


On Thu, Oct 29, 2020 at 8:29 PM Ed Bueler  wrote:

> All --
>
> SIAM Press just published my new book "PETSc for Partial Differential
> Equations: Numerical Solutions in C and Python":
>
>   https://my.siam.org/Store/Product/viewproduct/?ProductId=32850137
>
> The book is available both as a paperback and an e-book with working
> links.  A SIAM member discount is available, of course.
>
> This book is a genuine introduction which does not assume you have used
> PETSc before, and which should make sense even if your differential
> equations knowledge is basic.  The prerequisites are a bit of programming
> in C and a bit of numerical linear algebra, roughly like the main ideas of
> Trefethen and Bau, but even that is reviewed and summarized.  I've made an
> effort to introduce discretizations from the beginning, especially finite
> differences and elements.
>
> The book is based on a collection of example programs at
> https://github.com/bueler/p4pdes.  Most of these codes call PETSc
> directly through the C API, but the last two chapters have Python codes
> using UFL and Firedrake.  Nonetheless the book contains ideas, mathematical
> and computational; it complements, but does not replace, the PETSc User's
> Manual and the tutorial examples in the PETSc source.  Concepts are
> explained and illustrated, with sufficient context to facilitate further
> development. Performance (optimality) and parallel scalability are the
> primary goals, so preconditioners including multigrid are central threads,
> and run-time solver options are explored in both the text and the exercises.
>
> Here is the place to appreciate the usual PETSc suspects for their
> comments on drafts, and help in writing this book: Barry, Jed, Matt, Dave,
> Rich, Lois, Patrick, Mark, Satish, David K., and many others.  Also let me
> say that SIAM Press has nothing but professionals who are nice to work with
> too; send them your book idea!
>
> Ed
>
> --
> Ed Bueler
> Dept of Mathematics and Statistics
> University of Alaska Fairbanks
> Fairbanks, AK 99775-6660
> 306C Chapman
>