Re: [petsc-dev] including PETSc4py (and Tao4py) inside PETSc repository

2015-10-26 Thread Lisandro Dalcin 
On 26 October 2015 at 01:44, Barry Smith  wrote:
> Maybe someday git will have additional features that allow multiple repos to 
> share things like branches, commits, other stuff? so that having all these 
> packages in different repos is possible, but for now it appears having one 
> repo is just so much dang more convenient.

A final note regarding the subject of this thread: tao4py is part of
petsc4py since TAO was incorporated to PETSc.


-- 
Lisandro Dalcin

Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Numerical Porous Media Center (NumPor)
King Abdullah University of Science and Technology (KAUST)
http://numpor.kaust.edu.sa/

4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 4332
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa

Office Phone: +966 12 808-0459

Re: [petsc-dev] including PETSc4py (and Tao4py) inside PETSc repository

2015-10-25 Thread Lisandro Dalcin 
On 23 October 2015 at 06:30, Barry Smith  wrote:
>
>> On Oct 22, 2015, at 10:24 PM, Satish Balay  wrote:
>>
>> As I said - depending on the precondition for how tightly petsc &
>> petsc4py should be synced - we can choose the appropriate model.
>>
>> [for the metioned precondition - I was suggestion the model].
>>
>> As you say - the initilal phrasing is wrong - so the model is not
>> suitable.
>>
>> A single repo does make it easy for 'all feature branches' to be in
>> sync..
>>
>> In that model with-petsc4py=1 would be the default. However anyone
>> changing the petsc UI would also have to update petsc4py with the
>> change.. [and Lisandro says - that change could be buggy - and he
>> would like to review it -before its applied]
>
>Just like Jed was going to review everything anybody put into PETSc before 
> it got into master. That is just unrealistic and won't last long. Better to 
> just have a good test suite.
>

OK, Barry, I very much understand all your concerns. My opposition to
incorporate petsc4py within petsc is not strong. I agree maintenance
and contribution will be much easier, my concern is just about adding
more and more stuff to the PETSc repo  year over year. If for the case
of petsc4py you think that is not a big issue, then I have to trust
you, it is not easy to catch you on the wrong side ;-).

BTW, I no longer care about keeping petsc4py compatible with previous
PETSc releases. While such approach is workable for pure C code (like
in PetIGA), things get complicated when you add Cython to the stack.

So, let's go ahead and do the merge.




-- 
Lisandro Dalcin

Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Numerical Porous Media Center (NumPor)
King Abdullah University of Science and Technology (KAUST)
http://numpor.kaust.edu.sa/

4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 4332
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa

Office Phone: +966 12 808-0459

Re: [petsc-dev] including PETSc4py (and Tao4py) inside PETSc repository

2015-10-25 Thread Barry Smith 

  Maybe someday git will have additional features that allow multiple repos to 
share things like branches, commits, other stuff? so that having all these 
packages in different repos is possible, but for now it appears having one repo 
is just so much dang more convenient.


> On Oct 25, 2015, at 3:12 AM, Lisandro Dalcin  wrote:
> 
> On 23 October 2015 at 06:30, Barry Smith  wrote:
>> 
>>> On Oct 22, 2015, at 10:24 PM, Satish Balay  wrote:
>>> 
>>> As I said - depending on the precondition for how tightly petsc &
>>> petsc4py should be synced - we can choose the appropriate model.
>>> 
>>> [for the metioned precondition - I was suggestion the model].
>>> 
>>> As you say - the initilal phrasing is wrong - so the model is not
>>> suitable.
>>> 
>>> A single repo does make it easy for 'all feature branches' to be in
>>> sync..
>>> 
>>> In that model with-petsc4py=1 would be the default. However anyone
>>> changing the petsc UI would also have to update petsc4py with the
>>> change.. [and Lisandro says - that change could be buggy - and he
>>> would like to review it -before its applied]
>> 
>>   Just like Jed was going to review everything anybody put into PETSc before 
>> it got into master. That is just unrealistic and won't last long. Better to 
>> just have a good test suite.
>> 
> 
> OK, Barry, I very much understand all your concerns. My opposition to
> incorporate petsc4py within petsc is not strong. I agree maintenance
> and contribution will be much easier, my concern is just about adding
> more and more stuff to the PETSc repo  year over year. If for the case
> of petsc4py you think that is not a big issue, then I have to trust
> you, it is not easy to catch you on the wrong side ;-).
> 
> BTW, I no longer care about keeping petsc4py compatible with previous
> PETSc releases. While such approach is workable for pure C code (like
> in PetIGA), things get complicated when you add Cython to the stack.
> 
> So, let's go ahead and do the merge.
> 
> 
> 
> 
> -- 
> Lisandro Dalcin
> 
> Research Scientist
> Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
> Numerical Porous Media Center (NumPor)
> King Abdullah University of Science and Technology (KAUST)
> http://numpor.kaust.edu.sa/
> 
> 4700 King Abdullah University of Science and Technology
> al-Khawarizmi Bldg (Bldg 1), Office # 4332
> Thuwal 23955-6900, Kingdom of Saudi Arabia
> http://www.kaust.edu.sa
> 
> Office Phone: +966 12 808-0459

Re: [petsc-dev] including PETSc4py (and Tao4py) inside PETSc repository

2015-10-22 Thread Barry Smith 

   Lisandro,

You never responded to this. I assume it was because you did not like the 
idea? This is not an attempt to take control of petsc4py or to take credit away 
from petsc4py from you. Petsc4py is your package and you will always be the one 
who deserves the credit.

The problem is that mutual efficient development of very intertwined 
packages is difficult if they are in different respositories. For example 
petsc4py has been out of date with petsc master for many weeks now because it 
is too much effort to update the petsc4py repository each time we make some 
change in PETSc. Thus it becomes a burden on you to go in every once in a while 
and fix up petsc4py. With one repository changes would happen in both PETSc 
source and petsc4py source at the same time and tests would detect problems 
immediately.

   Thoughts?

   Barry



> On Sep 22, 2015, at 3:39 PM, Barry Smith  wrote:
> 
> 
>   Lisandro,
> 
>Now that everyone is using git and knows branches well can we move 
> PETSc4py inside the PETSc repository as was done a couple of years ago with 
> BuildSystem. Now when changes/additions are made to PETSc there is a slow 
> inefficient, often forgotten manual process of bringing them over to 
> petsc4py. If we put them all in one repository the updates happen quickly and 
> far more efficiently, as would testing. We'll also get the ability to do 
> bi-section.
> 
>   I realize you want a PETSc4py release to be compatible with previous 
> versions of PETSc; this property could still remain and we could easily have 
> a tool that "pulls out" the petsc4py release material from the petsc 
> repository.
> 
>   I think once we make this change we'll wonder why we didn't do it log ago; 
> just as when we changed BuildSystem.  Thoughts?
> 
>   Thanks
> 
>Barry
>

Re: [petsc-dev] including PETSc4py (and Tao4py) inside PETSc repository

2015-10-22 Thread Barry Smith 

  Lisandro,

   It is tested nightly but there is apparently currently no email sent on 
failure. See the bottom of, for example, 
http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2015/10/22/make_next_arch-linux-pkgs-opt_crank.log

   I think what should happen is the petsc4py log file should be published to 
the web (like the other log files) and a script look through it and decide to 
send email if it detects a problem. Like we currently have for the other log 
files. Someone has to dig into the nightly test infrastructure to figure out 
where the new logic needs to go. Currently Satish and Karl understand that 
infrastructure the best.

  Barry





> On Oct 22, 2015, at 6:44 PM, Lisandro Dalcin  wrote:
> 
> On 22 October 2015 at 23:20, Barry Smith  wrote:
>> 
>>   Lisandro,
>> 
>>You never responded to this. I assume it was because you did not like the 
>> idea? This is not an attempt to take control of petsc4py or to take credit 
>> away from petsc4py from you. Petsc4py is your package and you will always be 
>> the one who deserves the credit.
>> 
> 
> Oh, sorry! I missed this email.
> 
> 
>>The problem is that mutual efficient development of very intertwined 
>> packages is difficult if they are in different respositories. For example 
>> petsc4py has been out of date with petsc master for many weeks now because 
>> it is too much effort to update the petsc4py repository each time we make 
>> some change in PETSc. Thus it becomes a burden on you to go in every once in 
>> a while and fix up petsc4py. With one repository changes would happen in 
>> both PETSc source and petsc4py source at the same time and tests would 
>> detect problems immediately.
>> 
>>   Thoughts?
>> 
> 
> I general terms, I prefer to keep things separate as much a possible.
> IMHO, a much important step would be to incorporate petsc4py to the
> nightly tests, clearly separating the petsc and petsc4py tests runs. I
> would love to get an email every time petsc4py nightly breaks, this
> way I would be able to push fixes within a day or two, say.
> Maintaining petsc4py requires knowledge of PETSc+Python+Cython, so
> ultimately I'm the one in best position to do such work, if you guys
> start adding commits to petsc4py withing the main petsc repo, I'm
> likely going to miss these changes. For example, today Michael Lange
> made some changes, I got an email from Bitbucket, then did a review
> and spotted a few issues.
> 
> My hesitation to incorporate petsc4py within petsc repo is, in
> Python's [and linear algebra :-)] Zen : "Sparse is better than dense".
> If we continue adding stuff to core PETSc, at same point we will endup
> like Trilinos ( DON'T MISS the sencond entry in the FAQ:
> https://trilinos.org/download/public-git-repository/).
> 
> All that being said, I don't have any problem at all adding petsc4py
> to the main PETSc repository. But I still think this is not a good
> idea, it will not solve all the problems. Nightly tests with automatic
> notifications to me about breakages is perhaps is all what we need to
> keep petsc4py in close sync with petsc/master. What about trying this
> first to see how it goes?
> 
> 
> PS: Please understand my position has nothing to do with credits about
> the project or ego-related crap. Actually, your proposal would mean
> less responsibility and maintenance work for me. But I don't think it
> is a move in the right direction.
> 
> 
> -- 
> Lisandro Dalcin
> 
> Research Scientist
> Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
> Numerical Porous Media Center (NumPor)
> King Abdullah University of Science and Technology (KAUST)
> http://numpor.kaust.edu.sa/
> 
> 4700 King Abdullah University of Science and Technology
> al-Khawarizmi Bldg (Bldg 1), Office # 4332
> Thuwal 23955-6900, Kingdom of Saudi Arabia
> http://www.kaust.edu.sa
> 
> Office Phone: +966 12 808-0459

Re: [petsc-dev] including PETSc4py (and Tao4py) inside PETSc repository

2015-10-22 Thread Lisandro Dalcin 
On 22 October 2015 at 23:20, Barry Smith  wrote:
>
>Lisandro,
>
> You never responded to this. I assume it was because you did not like the 
> idea? This is not an attempt to take control of petsc4py or to take credit 
> away from petsc4py from you. Petsc4py is your package and you will always be 
> the one who deserves the credit.
>

Oh, sorry! I missed this email.


> The problem is that mutual efficient development of very intertwined 
> packages is difficult if they are in different respositories. For example 
> petsc4py has been out of date with petsc master for many weeks now because it 
> is too much effort to update the petsc4py repository each time we make some 
> change in PETSc. Thus it becomes a burden on you to go in every once in a 
> while and fix up petsc4py. With one repository changes would happen in both 
> PETSc source and petsc4py source at the same time and tests would detect 
> problems immediately.
>
>Thoughts?
>

I general terms, I prefer to keep things separate as much a possible.
IMHO, a much important step would be to incorporate petsc4py to the
nightly tests, clearly separating the petsc and petsc4py tests runs. I
would love to get an email every time petsc4py nightly breaks, this
way I would be able to push fixes within a day or two, say.
Maintaining petsc4py requires knowledge of PETSc+Python+Cython, so
ultimately I'm the one in best position to do such work, if you guys
start adding commits to petsc4py withing the main petsc repo, I'm
likely going to miss these changes. For example, today Michael Lange
made some changes, I got an email from Bitbucket, then did a review
and spotted a few issues.

My hesitation to incorporate petsc4py within petsc repo is, in
Python's [and linear algebra :-)] Zen : "Sparse is better than dense".
If we continue adding stuff to core PETSc, at same point we will endup
like Trilinos ( DON'T MISS the sencond entry in the FAQ:
https://trilinos.org/download/public-git-repository/).

All that being said, I don't have any problem at all adding petsc4py
to the main PETSc repository. But I still think this is not a good
idea, it will not solve all the problems. Nightly tests with automatic
notifications to me about breakages is perhaps is all what we need to
keep petsc4py in close sync with petsc/master. What about trying this
first to see how it goes?


PS: Please understand my position has nothing to do with credits about
the project or ego-related crap. Actually, your proposal would mean
less responsibility and maintenance work for me. But I don't think it
is a move in the right direction.


-- 
Lisandro Dalcin

Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Numerical Porous Media Center (NumPor)
King Abdullah University of Science and Technology (KAUST)
http://numpor.kaust.edu.sa/

4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 4332
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa

Office Phone: +966 12 808-0459

Re: [petsc-dev] including PETSc4py (and Tao4py) inside PETSc repository

2015-10-22 Thread Barry Smith 

> On Oct 22, 2015, at 7:28 PM, Satish Balay  wrote:
> 
> The current infrastruture tests --download-petsc4py in petsc 'next'
> and 'master' branches.
> 
> Wrt next - usually a change in a feature branch (when it gets merged
> to next) triggers this breakage.  And usual thing to do is to fix it
> in the feature branch (and merge to next again)
> 
> Wrt to petsc4py - this would mean updating petsc4py.py [in petsc] with
> the petsc4py repo gitcommit [optionally also tarball ] that
> corresponds to the fix. [one way to handle this on petsc4py repo is to
> create a feature branch for this fix]

  Yeah this is nuts, having to maintaining matching branches between petsc and 
petsc4py and merge them at the same time into next and then master. Clearly no 
one will ever do this and hence petsc4py will always be out of step from PETSc. 
I do really believe that incorporating petsc4py into PETSc would just make all 
these difficulties go away. For example maintaining Tao is a breeze now since 
it is updated in the same branches as PETSc.

  Barry

> 
> When this (petsc) feature branch is merged into master - the
> corresponding feature branch on the petsc4py side can be merged into
> petsc4py master [if petsc4py-master is to be in sync with
> petsc-master].
> 
> [if there are multiple feature branches on the petsc side requiring
> corresponding petsc4py changes - somehow that has to be handled]
> 
> Perhaps testing petsc4py via petsc infrastructure [using
> --download-petsc4py] is not the appropriate approach?
> 
> Or we should do this only on master - not next?  [or just have a
> single --download-petsc4py test - thats not mixed with other tests? -
> so its breakage is not critical for other things]
> 
> Also we don't currently run any petsc4py tests. [we just test if it
> compiles or not]
> 
> Satish
> 
> On Thu, 22 Oct 2015, Barry Smith wrote:
> 
>> 
>>  Lisandro,
>> 
>>   It is tested nightly but there is apparently currently no email sent on 
>> failure. See the bottom of, for example, 
>> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2015/10/22/make_next_arch-linux-pkgs-opt_crank.log
>> 
>>   I think what should happen is the petsc4py log file should be published to 
>> the web (like the other log files) and a script look through it and decide 
>> to send email if it detects a problem. Like we currently have for the other 
>> log files. Someone has to dig into the nightly test infrastructure to figure 
>> out where the new logic needs to go. Currently Satish and Karl understand 
>> that infrastructure the best.
>> 
>>  Barry
>> 
>> 
>> 
>> 
>> 
>>> On Oct 22, 2015, at 6:44 PM, Lisandro Dalcin  wrote:
>>> 
>>> On 22 October 2015 at 23:20, Barry Smith  wrote:
 
  Lisandro,
 
   You never responded to this. I assume it was because you did not like 
 the idea? This is not an attempt to take control of petsc4py or to take 
 credit away from petsc4py from you. Petsc4py is your package and you will 
 always be the one who deserves the credit.
 
>>> 
>>> Oh, sorry! I missed this email.
>>> 
>>> 
   The problem is that mutual efficient development of very intertwined 
 packages is difficult if they are in different respositories. For example 
 petsc4py has been out of date with petsc master for many weeks now because 
 it is too much effort to update the petsc4py repository each time we make 
 some change in PETSc. Thus it becomes a burden on you to go in every once 
 in a while and fix up petsc4py. With one repository changes would happen 
 in both PETSc source and petsc4py source at the same time and tests would 
 detect problems immediately.
 
  Thoughts?
 
>>> 
>>> I general terms, I prefer to keep things separate as much a possible.
>>> IMHO, a much important step would be to incorporate petsc4py to the
>>> nightly tests, clearly separating the petsc and petsc4py tests runs. I
>>> would love to get an email every time petsc4py nightly breaks, this
>>> way I would be able to push fixes within a day or two, say.
>>> Maintaining petsc4py requires knowledge of PETSc+Python+Cython, so
>>> ultimately I'm the one in best position to do such work, if you guys
>>> start adding commits to petsc4py withing the main petsc repo, I'm
>>> likely going to miss these changes. For example, today Michael Lange
>>> made some changes, I got an email from Bitbucket, then did a review
>>> and spotted a few issues.
>>> 
>>> My hesitation to incorporate petsc4py within petsc repo is, in
>>> Python's [and linear algebra :-)] Zen : "Sparse is better than dense".
>>> If we continue adding stuff to core PETSc, at same point we will endup
>>> like Trilinos ( DON'T MISS the sencond entry in the FAQ:
>>> https://trilinos.org/download/public-git-repository/).
>>> 
>>> All that being said, I don't have any problem at all adding petsc4py
>>> to the main PETSc repository. But I still think this is not a good

Re: [petsc-dev] including PETSc4py (and Tao4py) inside PETSc repository

2015-10-22 Thread Barry Smith 

  The model we adopt also depends on how cutting edge petsc4py users want to be 
with petsc. For example if people want to use dmplex with petsc4py then the two 
packages have to move very closely in sync.

> On Oct 22, 2015, at 9:11 PM, Satish Balay  wrote:
> 
> On Thu, 22 Oct 2015, Barry Smith wrote:
> 
>> 
>>> On Oct 22, 2015, at 7:28 PM, Satish Balay  wrote:
>>> 
>>> The current infrastruture tests --download-petsc4py in petsc 'next'
>>> and 'master' branches.
>>> 
>>> Wrt next - usually a change in a feature branch (when it gets merged
>>> to next) triggers this breakage.  And usual thing to do is to fix it
>>> in the feature branch (and merge to next again)
>>> 
>>> Wrt to petsc4py - this would mean updating petsc4py.py [in petsc] with
>>> the petsc4py repo gitcommit [optionally also tarball ] that
>>> corresponds to the fix. [one way to handle this on petsc4py repo is to
>>> create a feature branch for this fix]
>> 
>>  Yeah this is nuts, having to maintaining matching branches between petsc 
>> and petsc4py and merge them at the same time into next and then master. 
>> Clearly no one will ever do this and hence petsc4py will always be out of 
>> step from PETSc. I do really believe that incorporating petsc4py into PETSc 
>> would just make all these difficulties go away. For example maintaining Tao 
>> is a breeze now since it is updated in the same branches as PETSc.
>> 
> 
> Perhaps I should rephrase the issue in the following way:
> 
> We should decide on what level petsc4py and petsc should be synced.
> [and the testsuite should be modified appropriately for that]
> 
> Currently its done at next, master branchs. For this to work - the
> above - previously mentioned workflow is required. [and for this
> workflow - a single repo does makes thing easier]
> 
> We could say sync should happen only between petsc4py-master &
> petsc-master. [so issues - if any in feature branches/next wont't be
> dealt with - and its ok for the master branches to be out of sync for
> a brief time for the fix to propergate].
> 
> For this - we would have a different workflow [wrt syncing petsc4py
> wrt petsc - and a slightly different test suite - testing only
> master. [We could continue having the curent test suite - but ignore
> errors in next].  And e-mails to Lisandro will trigger only on
> petsc4py errors on master.  Lisandro would then fix petsc4py (master)
> - and then update petsc4py.py in petsc-master.
> 
> Or perhaps some other mode is preferable..
> 
> Satish
> 
>>  Barry
>> 
>>> 
>>> When this (petsc) feature branch is merged into master - the
>>> corresponding feature branch on the petsc4py side can be merged into
>>> petsc4py master [if petsc4py-master is to be in sync with
>>> petsc-master].
>>> 
>>> [if there are multiple feature branches on the petsc side requiring
>>> corresponding petsc4py changes - somehow that has to be handled]
>>> 
>>> Perhaps testing petsc4py via petsc infrastructure [using
>>> --download-petsc4py] is not the appropriate approach?
>>> 
>>> Or we should do this only on master - not next?  [or just have a
>>> single --download-petsc4py test - thats not mixed with other tests? -
>>> so its breakage is not critical for other things]
>>> 
>>> Also we don't currently run any petsc4py tests. [we just test if it
>>> compiles or not]
>>> 
>>> Satish
>>> 
>>> On Thu, 22 Oct 2015, Barry Smith wrote:
>>> 
 
 Lisandro,
 
  It is tested nightly but there is apparently currently no email sent on 
 failure. See the bottom of, for example, 
 http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2015/10/22/make_next_arch-linux-pkgs-opt_crank.log
 
  I think what should happen is the petsc4py log file should be published 
 to the web (like the other log files) and a script look through it and 
 decide to send email if it detects a problem. Like we currently have for 
 the other log files. Someone has to dig into the nightly test 
 infrastructure to figure out where the new logic needs to go. Currently 
 Satish and Karl understand that infrastructure the best.
 
 Barry
 
 
 
 
 
> On Oct 22, 2015, at 6:44 PM, Lisandro Dalcin  wrote:
> 
> On 22 October 2015 at 23:20, Barry Smith  wrote:
>> 
>> Lisandro,
>> 
>>  You never responded to this. I assume it was because you did not like 
>> the idea? This is not an attempt to take control of petsc4py or to take 
>> credit away from petsc4py from you. Petsc4py is your package and you 
>> will always be the one who deserves the credit.
>> 
> 
> Oh, sorry! I missed this email.
> 
> 
>>  The problem is that mutual efficient development of very intertwined 
>> packages is difficult if they are in different respositories. For 
>> example petsc4py has been out of date with petsc master for many weeks 
>> now because it is too much

Re: [petsc-dev] including PETSc4py (and Tao4py) inside PETSc repository

2015-10-22 Thread Barry Smith 

> On Oct 22, 2015, at 10:24 PM, Satish Balay  wrote:
> 
> As I said - depending on the precondition for how tightly petsc &
> petsc4py should be synced - we can choose the appropriate model.
> 
> [for the metioned precondition - I was suggestion the model].
> 
> As you say - the initilal phrasing is wrong - so the model is not
> suitable.
> 
> A single repo does make it easy for 'all feature branches' to be in
> sync..
> 
> In that model with-petsc4py=1 would be the default. However anyone
> changing the petsc UI would also have to update petsc4py with the
> change.. [and Lisandro says - that change could be buggy - and he
> would like to review it -before its applied]

   Just like Jed was going to review everything anybody put into PETSc before 
it got into master. That is just unrealistic and won't last long. Better to 
just have a good test suite.

> 
> For 2 repos model - perhaps sync at 'master' might work.
> 
> Satish
> 
> On Thu, 22 Oct 2015, Barry Smith wrote:
> 
>> 
>>> On Oct 22, 2015, at 9:53 PM, Satish Balay  wrote:
>>> 
>>> If dmplex with petsc4py is the primary use case [and assuming the
>>> usage is primarily --download-petsc4py] - we chould use the first
>>> model I mentioned - and that should not be too difficult.
>>> 
>>> The workflow would be something like:
>>> 
>>> - Matt would do the updates in petsc/dmplex feature branch & petsc4py
>>> dmplex branches simultaneosusly.
>> 
>>   That's nuts; Matt is a college professor he doesn't have time to keep two 
>> branches in sync for no good reason when they could be in the same 
>> repository.
>> 
>>   Plus other people will be making other changes in PETSc that break petsc4py
>> 
>>   This discussion has shown to me that keeping petsc and petsc4py in 
>> different repositories is just silly and unproductive.
>>> 
>>> - And would always use --download-petsc4py for all his build tests
>>> [we'll have to fix auto-update for --download-pkg for git version of
>>> pkg]
>>> 
>>> - Any isues that come up - Matt will fix in petsc4py/dmplex and update
>>> petsc4py [before this branch is ever merged to next]
>>> 
>>> So after next testing - a merge to master - Matt could issue a pull
>>> requst to Lisandro [or leave it for Lisandro to figureout when
>>> petsc4py-master gets updated with the fix].
>>> 
>>> The update would involve merging/updating petsc4py-master [and and
>>> update to petsc4py.py in petsc-master]
>>> 
>>> [even if this petsc4py-master update doesn't happen immediately] it
>>> won't affect petsc4py/dmplex users as they would be using
>>> --download-petsc4py [which will use the working petsc4py.commitid set
>>> my Matt anyway]
>>> 
>>> If the usage is not via --download-petsc4py - then there could be a
>>> more appropriate workflow for that.. [and a different testsuite to
>>> match that usage..]
>>> 
>>> Satish
>>> 
>>> On Thu, 22 Oct 2015, Barry Smith wrote:
>>> 
 
 The model we adopt also depends on how cutting edge petsc4py users want to 
 be with petsc. For example if people want to use dmplex with petsc4py then 
 the two packages have to move very closely in sync.
 
> On Oct 22, 2015, at 9:11 PM, Satish Balay  wrote:
> 
> On Thu, 22 Oct 2015, Barry Smith wrote:
> 
>> 
>>> On Oct 22, 2015, at 7:28 PM, Satish Balay  wrote:
>>> 
>>> The current infrastruture tests --download-petsc4py in petsc 'next'
>>> and 'master' branches.
>>> 
>>> Wrt next - usually a change in a feature branch (when it gets merged
>>> to next) triggers this breakage.  And usual thing to do is to fix it
>>> in the feature branch (and merge to next again)
>>> 
>>> Wrt to petsc4py - this would mean updating petsc4py.py [in petsc] with
>>> the petsc4py repo gitcommit [optionally also tarball ] that
>>> corresponds to the fix. [one way to handle this on petsc4py repo is to
>>> create a feature branch for this fix]
>> 
>> Yeah this is nuts, having to maintaining matching branches between petsc 
>> and petsc4py and merge them at the same time into next and then master. 
>> Clearly no one will ever do this and hence petsc4py will always be out 
>> of step from PETSc. I do really believe that incorporating petsc4py into 
>> PETSc would just make all these difficulties go away. For example 
>> maintaining Tao is a breeze now since it is updated in the same branches 
>> as PETSc.
>> 
> 
> Perhaps I should rephrase the issue in the following way:
> 
> We should decide on what level petsc4py and petsc should be synced.
> [and the testsuite should be modified appropriately for that]
> 
> Currently its done at next, master branchs. For this to work - the
> above - previously mentioned workflow is required. [and for this
> workflow - a single repo does makes thing easier]
> 
> We could say sync should happen only between

Re: [petsc-dev] including PETSc4py (and Tao4py) inside PETSc repository

2015-10-22 Thread Satish Balay 
On Thu, 22 Oct 2015, Barry Smith wrote:

> 
> > On Oct 22, 2015, at 7:28 PM, Satish Balay  wrote:
> > 
> > The current infrastruture tests --download-petsc4py in petsc 'next'
> > and 'master' branches.
> > 
> > Wrt next - usually a change in a feature branch (when it gets merged
> > to next) triggers this breakage.  And usual thing to do is to fix it
> > in the feature branch (and merge to next again)
> > 
> > Wrt to petsc4py - this would mean updating petsc4py.py [in petsc] with
> > the petsc4py repo gitcommit [optionally also tarball ] that
> > corresponds to the fix. [one way to handle this on petsc4py repo is to
> > create a feature branch for this fix]
> 
>   Yeah this is nuts, having to maintaining matching branches between petsc 
> and petsc4py and merge them at the same time into next and then master. 
> Clearly no one will ever do this and hence petsc4py will always be out of 
> step from PETSc. I do really believe that incorporating petsc4py into PETSc 
> would just make all these difficulties go away. For example maintaining Tao 
> is a breeze now since it is updated in the same branches as PETSc.
> 

Perhaps I should rephrase the issue in the following way:

We should decide on what level petsc4py and petsc should be synced.
[and the testsuite should be modified appropriately for that]

Currently its done at next, master branchs. For this to work - the
above - previously mentioned workflow is required. [and for this
workflow - a single repo does makes thing easier]

We could say sync should happen only between petsc4py-master &
petsc-master. [so issues - if any in feature branches/next wont't be
dealt with - and its ok for the master branches to be out of sync for
a brief time for the fix to propergate].

For this - we would have a different workflow [wrt syncing petsc4py
wrt petsc - and a slightly different test suite - testing only
master. [We could continue having the curent test suite - but ignore
errors in next].  And e-mails to Lisandro will trigger only on
petsc4py errors on master.  Lisandro would then fix petsc4py (master)
- and then update petsc4py.py in petsc-master.

Or perhaps some other mode is preferable..

Satish

>   Barry
> 
> > 
> > When this (petsc) feature branch is merged into master - the
> > corresponding feature branch on the petsc4py side can be merged into
> > petsc4py master [if petsc4py-master is to be in sync with
> > petsc-master].
> > 
> > [if there are multiple feature branches on the petsc side requiring
> > corresponding petsc4py changes - somehow that has to be handled]
> > 
> > Perhaps testing petsc4py via petsc infrastructure [using
> > --download-petsc4py] is not the appropriate approach?
> > 
> > Or we should do this only on master - not next?  [or just have a
> > single --download-petsc4py test - thats not mixed with other tests? -
> > so its breakage is not critical for other things]
> > 
> > Also we don't currently run any petsc4py tests. [we just test if it
> > compiles or not]
> > 
> > Satish
> > 
> > On Thu, 22 Oct 2015, Barry Smith wrote:
> > 
> >> 
> >>  Lisandro,
> >> 
> >>   It is tested nightly but there is apparently currently no email sent on 
> >> failure. See the bottom of, for example, 
> >> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2015/10/22/make_next_arch-linux-pkgs-opt_crank.log
> >> 
> >>   I think what should happen is the petsc4py log file should be published 
> >> to the web (like the other log files) and a script look through it and 
> >> decide to send email if it detects a problem. Like we currently have for 
> >> the other log files. Someone has to dig into the nightly test 
> >> infrastructure to figure out where the new logic needs to go. Currently 
> >> Satish and Karl understand that infrastructure the best.
> >> 
> >>  Barry
> >> 
> >> 
> >> 
> >> 
> >> 
> >>> On Oct 22, 2015, at 6:44 PM, Lisandro Dalcin  wrote:
> >>> 
> >>> On 22 October 2015 at 23:20, Barry Smith  wrote:
>  
>   Lisandro,
>  
>    You never responded to this. I assume it was because you did not like 
>  the idea? This is not an attempt to take control of petsc4py or to take 
>  credit away from petsc4py from you. Petsc4py is your package and you 
>  will always be the one who deserves the credit.
>  
> >>> 
> >>> Oh, sorry! I missed this email.
> >>> 
> >>> 
>    The problem is that mutual efficient development of very intertwined 
>  packages is difficult if they are in different respositories. For 
>  example petsc4py has been out of date with petsc master for many weeks 
>  now because it is too much effort to update the petsc4py repository each 
>  time we make some change in PETSc. Thus it becomes a burden on you to go 
>  in every once in a while and fix up petsc4py. With one repository 
>  changes would happen in both PETSc source and petsc4py source at the 
>  same time and tests would detect problems immediately.

Re: [petsc-dev] including PETSc4py (and Tao4py) inside PETSc repository

2015-10-22 Thread Satish Balay 
As I said - depending on the precondition for how tightly petsc &
petsc4py should be synced - we can choose the appropriate model.

[for the metioned precondition - I was suggestion the model].

As you say - the initilal phrasing is wrong - so the model is not
suitable.

A single repo does make it easy for 'all feature branches' to be in
sync..

In that model with-petsc4py=1 would be the default. However anyone
changing the petsc UI would also have to update petsc4py with the
change.. [and Lisandro says - that change could be buggy - and he
would like to review it -before its applied]

For 2 repos model - perhaps sync at 'master' might work.

Satish

On Thu, 22 Oct 2015, Barry Smith wrote:

> 
> > On Oct 22, 2015, at 9:53 PM, Satish Balay  wrote:
> > 
> > If dmplex with petsc4py is the primary use case [and assuming the
> > usage is primarily --download-petsc4py] - we chould use the first
> > model I mentioned - and that should not be too difficult.
> > 
> > The workflow would be something like:
> > 
> > - Matt would do the updates in petsc/dmplex feature branch & petsc4py
> >  dmplex branches simultaneosusly.
> 
>That's nuts; Matt is a college professor he doesn't have time to keep two 
> branches in sync for no good reason when they could be in the same repository.
> 
>Plus other people will be making other changes in PETSc that break petsc4py
> 
>This discussion has shown to me that keeping petsc and petsc4py in 
> different repositories is just silly and unproductive.
> > 
> > - And would always use --download-petsc4py for all his build tests
> > [we'll have to fix auto-update for --download-pkg for git version of
> > pkg]
> > 
> > - Any isues that come up - Matt will fix in petsc4py/dmplex and update
> >  petsc4py [before this branch is ever merged to next]
> > 
> > So after next testing - a merge to master - Matt could issue a pull
> > requst to Lisandro [or leave it for Lisandro to figureout when
> > petsc4py-master gets updated with the fix].
> > 
> > The update would involve merging/updating petsc4py-master [and and
> > update to petsc4py.py in petsc-master]
> > 
> > [even if this petsc4py-master update doesn't happen immediately] it
> > won't affect petsc4py/dmplex users as they would be using
> > --download-petsc4py [which will use the working petsc4py.commitid set
> > my Matt anyway]
> > 
> > If the usage is not via --download-petsc4py - then there could be a
> > more appropriate workflow for that.. [and a different testsuite to
> > match that usage..]
> > 
> > Satish
> > 
> > On Thu, 22 Oct 2015, Barry Smith wrote:
> > 
> >> 
> >>  The model we adopt also depends on how cutting edge petsc4py users want 
> >> to be with petsc. For example if people want to use dmplex with petsc4py 
> >> then the two packages have to move very closely in sync.
> >> 
> >>> On Oct 22, 2015, at 9:11 PM, Satish Balay  wrote:
> >>> 
> >>> On Thu, 22 Oct 2015, Barry Smith wrote:
> >>> 
>  
> > On Oct 22, 2015, at 7:28 PM, Satish Balay  wrote:
> > 
> > The current infrastruture tests --download-petsc4py in petsc 'next'
> > and 'master' branches.
> > 
> > Wrt next - usually a change in a feature branch (when it gets merged
> > to next) triggers this breakage.  And usual thing to do is to fix it
> > in the feature branch (and merge to next again)
> > 
> > Wrt to petsc4py - this would mean updating petsc4py.py [in petsc] with
> > the petsc4py repo gitcommit [optionally also tarball ] that
> > corresponds to the fix. [one way to handle this on petsc4py repo is to
> > create a feature branch for this fix]
>  
>  Yeah this is nuts, having to maintaining matching branches between petsc 
>  and petsc4py and merge them at the same time into next and then master. 
>  Clearly no one will ever do this and hence petsc4py will always be out 
>  of step from PETSc. I do really believe that incorporating petsc4py into 
>  PETSc would just make all these difficulties go away. For example 
>  maintaining Tao is a breeze now since it is updated in the same branches 
>  as PETSc.
>  
> >>> 
> >>> Perhaps I should rephrase the issue in the following way:
> >>> 
> >>> We should decide on what level petsc4py and petsc should be synced.
> >>> [and the testsuite should be modified appropriately for that]
> >>> 
> >>> Currently its done at next, master branchs. For this to work - the
> >>> above - previously mentioned workflow is required. [and for this
> >>> workflow - a single repo does makes thing easier]
> >>> 
> >>> We could say sync should happen only between petsc4py-master &
> >>> petsc-master. [so issues - if any in feature branches/next wont't be
> >>> dealt with - and its ok for the master branches to be out of sync for
> >>> a brief time for the fix to propergate].
> >>> 
> >>> For this - we would have a different workflow [wrt syncing petsc4py
> >>> wrt petsc - and a

Re: [petsc-dev] including PETSc4py (and Tao4py) inside PETSc repository

2015-10-22 Thread Barry Smith 

> On Oct 22, 2015, at 9:53 PM, Satish Balay  wrote:
> 
> If dmplex with petsc4py is the primary use case [and assuming the
> usage is primarily --download-petsc4py] - we chould use the first
> model I mentioned - and that should not be too difficult.
> 
> The workflow would be something like:
> 
> - Matt would do the updates in petsc/dmplex feature branch & petsc4py
>  dmplex branches simultaneosusly.

   That's nuts; Matt is a college professor he doesn't have time to keep two 
branches in sync for no good reason when they could be in the same repository.

   Plus other people will be making other changes in PETSc that break petsc4py

   This discussion has shown to me that keeping petsc and petsc4py in different 
repositories is just silly and unproductive.
> 
> - And would always use --download-petsc4py for all his build tests
> [we'll have to fix auto-update for --download-pkg for git version of
> pkg]
> 
> - Any isues that come up - Matt will fix in petsc4py/dmplex and update
>  petsc4py [before this branch is ever merged to next]
> 
> So after next testing - a merge to master - Matt could issue a pull
> requst to Lisandro [or leave it for Lisandro to figureout when
> petsc4py-master gets updated with the fix].
> 
> The update would involve merging/updating petsc4py-master [and and
> update to petsc4py.py in petsc-master]
> 
> [even if this petsc4py-master update doesn't happen immediately] it
> won't affect petsc4py/dmplex users as they would be using
> --download-petsc4py [which will use the working petsc4py.commitid set
> my Matt anyway]
> 
> If the usage is not via --download-petsc4py - then there could be a
> more appropriate workflow for that.. [and a different testsuite to
> match that usage..]
> 
> Satish
> 
> On Thu, 22 Oct 2015, Barry Smith wrote:
> 
>> 
>>  The model we adopt also depends on how cutting edge petsc4py users want to 
>> be with petsc. For example if people want to use dmplex with petsc4py then 
>> the two packages have to move very closely in sync.
>> 
>>> On Oct 22, 2015, at 9:11 PM, Satish Balay  wrote:
>>> 
>>> On Thu, 22 Oct 2015, Barry Smith wrote:
>>> 
 
> On Oct 22, 2015, at 7:28 PM, Satish Balay  wrote:
> 
> The current infrastruture tests --download-petsc4py in petsc 'next'
> and 'master' branches.
> 
> Wrt next - usually a change in a feature branch (when it gets merged
> to next) triggers this breakage.  And usual thing to do is to fix it
> in the feature branch (and merge to next again)
> 
> Wrt to petsc4py - this would mean updating petsc4py.py [in petsc] with
> the petsc4py repo gitcommit [optionally also tarball ] that
> corresponds to the fix. [one way to handle this on petsc4py repo is to
> create a feature branch for this fix]
 
 Yeah this is nuts, having to maintaining matching branches between petsc 
 and petsc4py and merge them at the same time into next and then master. 
 Clearly no one will ever do this and hence petsc4py will always be out of 
 step from PETSc. I do really believe that incorporating petsc4py into 
 PETSc would just make all these difficulties go away. For example 
 maintaining Tao is a breeze now since it is updated in the same branches 
 as PETSc.
 
>>> 
>>> Perhaps I should rephrase the issue in the following way:
>>> 
>>> We should decide on what level petsc4py and petsc should be synced.
>>> [and the testsuite should be modified appropriately for that]
>>> 
>>> Currently its done at next, master branchs. For this to work - the
>>> above - previously mentioned workflow is required. [and for this
>>> workflow - a single repo does makes thing easier]
>>> 
>>> We could say sync should happen only between petsc4py-master &
>>> petsc-master. [so issues - if any in feature branches/next wont't be
>>> dealt with - and its ok for the master branches to be out of sync for
>>> a brief time for the fix to propergate].
>>> 
>>> For this - we would have a different workflow [wrt syncing petsc4py
>>> wrt petsc - and a slightly different test suite - testing only
>>> master. [We could continue having the curent test suite - but ignore
>>> errors in next].  And e-mails to Lisandro will trigger only on
>>> petsc4py errors on master.  Lisandro would then fix petsc4py (master)
>>> - and then update petsc4py.py in petsc-master.
>>> 
>>> Or perhaps some other mode is preferable..
>>> 
>>> Satish
>>> 
 Barry
 
> 
> When this (petsc) feature branch is merged into master - the
> corresponding feature branch on the petsc4py side can be merged into
> petsc4py master [if petsc4py-master is to be in sync with
> petsc-master].
> 
> [if there are multiple feature branches on the petsc side requiring
> corresponding petsc4py changes - somehow that has to be handled]
> 
> Perhaps testing petsc4py via petsc infrastructure [using
> --download-petsc4py] is not the

Re: [petsc-dev] including PETSc4py (and Tao4py) inside PETSc repository

2015-10-22 Thread Satish Balay 
If dmplex with petsc4py is the primary use case [and assuming the
usage is primarily --download-petsc4py] - we chould use the first
model I mentioned - and that should not be too difficult.

The workflow would be something like:

- Matt would do the updates in petsc/dmplex feature branch & petsc4py
  dmplex branches simultaneosusly.

- And would always use --download-petsc4py for all his build tests
[we'll have to fix auto-update for --download-pkg for git version of
pkg]

- Any isues that come up - Matt will fix in petsc4py/dmplex and update
  petsc4py [before this branch is ever merged to next]

So after next testing - a merge to master - Matt could issue a pull
requst to Lisandro [or leave it for Lisandro to figureout when
petsc4py-master gets updated with the fix].

The update would involve merging/updating petsc4py-master [and and
update to petsc4py.py in petsc-master]

[even if this petsc4py-master update doesn't happen immediately] it
won't affect petsc4py/dmplex users as they would be using
--download-petsc4py [which will use the working petsc4py.commitid set
my Matt anyway]

If the usage is not via --download-petsc4py - then there could be a
more appropriate workflow for that.. [and a different testsuite to
match that usage..]

Satish

On Thu, 22 Oct 2015, Barry Smith wrote:

> 
>   The model we adopt also depends on how cutting edge petsc4py users want to 
> be with petsc. For example if people want to use dmplex with petsc4py then 
> the two packages have to move very closely in sync.
> 
> > On Oct 22, 2015, at 9:11 PM, Satish Balay  wrote:
> > 
> > On Thu, 22 Oct 2015, Barry Smith wrote:
> > 
> >> 
> >>> On Oct 22, 2015, at 7:28 PM, Satish Balay  wrote:
> >>> 
> >>> The current infrastruture tests --download-petsc4py in petsc 'next'
> >>> and 'master' branches.
> >>> 
> >>> Wrt next - usually a change in a feature branch (when it gets merged
> >>> to next) triggers this breakage.  And usual thing to do is to fix it
> >>> in the feature branch (and merge to next again)
> >>> 
> >>> Wrt to petsc4py - this would mean updating petsc4py.py [in petsc] with
> >>> the petsc4py repo gitcommit [optionally also tarball ] that
> >>> corresponds to the fix. [one way to handle this on petsc4py repo is to
> >>> create a feature branch for this fix]
> >> 
> >>  Yeah this is nuts, having to maintaining matching branches between petsc 
> >> and petsc4py and merge them at the same time into next and then master. 
> >> Clearly no one will ever do this and hence petsc4py will always be out of 
> >> step from PETSc. I do really believe that incorporating petsc4py into 
> >> PETSc would just make all these difficulties go away. For example 
> >> maintaining Tao is a breeze now since it is updated in the same branches 
> >> as PETSc.
> >> 
> > 
> > Perhaps I should rephrase the issue in the following way:
> > 
> > We should decide on what level petsc4py and petsc should be synced.
> > [and the testsuite should be modified appropriately for that]
> > 
> > Currently its done at next, master branchs. For this to work - the
> > above - previously mentioned workflow is required. [and for this
> > workflow - a single repo does makes thing easier]
> > 
> > We could say sync should happen only between petsc4py-master &
> > petsc-master. [so issues - if any in feature branches/next wont't be
> > dealt with - and its ok for the master branches to be out of sync for
> > a brief time for the fix to propergate].
> > 
> > For this - we would have a different workflow [wrt syncing petsc4py
> > wrt petsc - and a slightly different test suite - testing only
> > master. [We could continue having the curent test suite - but ignore
> > errors in next].  And e-mails to Lisandro will trigger only on
> > petsc4py errors on master.  Lisandro would then fix petsc4py (master)
> > - and then update petsc4py.py in petsc-master.
> > 
> > Or perhaps some other mode is preferable..
> > 
> > Satish
> > 
> >>  Barry
> >> 
> >>> 
> >>> When this (petsc) feature branch is merged into master - the
> >>> corresponding feature branch on the petsc4py side can be merged into
> >>> petsc4py master [if petsc4py-master is to be in sync with
> >>> petsc-master].
> >>> 
> >>> [if there are multiple feature branches on the petsc side requiring
> >>> corresponding petsc4py changes - somehow that has to be handled]
> >>> 
> >>> Perhaps testing petsc4py via petsc infrastructure [using
> >>> --download-petsc4py] is not the appropriate approach?
> >>> 
> >>> Or we should do this only on master - not next?  [or just have a
> >>> single --download-petsc4py test - thats not mixed with other tests? -
> >>> so its breakage is not critical for other things]
> >>> 
> >>> Also we don't currently run any petsc4py tests. [we just test if it
> >>> compiles or not]
> >>> 
> >>> Satish
> >>> 
> >>> On Thu, 22 Oct 2015, Barry Smith wrote:
> >>> 
>  
>  Lisandro,
>  
>   It is tested nightly but there is apparently currently no

[petsc-dev] including PETSc4py (and Tao4py) inside PETSc repository

2015-09-22 Thread Barry Smith 

   Lisandro,

Now that everyone is using git and knows branches well can we move PETSc4py 
inside the PETSc repository as was done a couple of years ago with BuildSystem. 
Now when changes/additions are made to PETSc there is a slow inefficient, often 
forgotten manual process of bringing them over to petsc4py. If we put them all 
in one repository the updates happen quickly and far more efficiently, as would 
testing. We'll also get the ability to do bi-section.

   I realize you want a PETSc4py release to be compatible with previous 
versions of PETSc; this property could still remain and we could easily have a 
tool that "pulls out" the petsc4py release material from the petsc repository.

   I think once we make this change we'll wonder why we didn't do it log ago; 
just as when we changed BuildSystem.  Thoughts?

   Thanks

Barry