If dmplex with petsc4py is the primary use case [and assuming the usage is primarily --download-petsc4py] - we chould use the first model I mentioned - and that should not be too difficult.
The workflow would be something like: - Matt would do the updates in petsc/dmplex feature branch & petsc4py dmplex branches simultaneosusly. - And would always use --download-petsc4py for all his build tests [we'll have to fix auto-update for --download-pkg for git version of pkg] - Any isues that come up - Matt will fix in petsc4py/dmplex and update petsc4py [before this branch is ever merged to next] So after next testing - a merge to master - Matt could issue a pull requst to Lisandro [or leave it for Lisandro to figureout when petsc4py-master gets updated with the fix]. The update would involve merging/updating petsc4py-master [and and update to petsc4py.py in petsc-master] [even if this petsc4py-master update doesn't happen immediately] it won't affect petsc4py/dmplex users as they would be using --download-petsc4py [which will use the working petsc4py.commitid set my Matt anyway] If the usage is not via --download-petsc4py - then there could be a more appropriate workflow for that.. [and a different testsuite to match that usage..] Satish On Thu, 22 Oct 2015, Barry Smith wrote: > > The model we adopt also depends on how cutting edge petsc4py users want to > be with petsc. For example if people want to use dmplex with petsc4py then > the two packages have to move very closely in sync. > > > On Oct 22, 2015, at 9:11 PM, Satish Balay <[email protected]> wrote: > > > > On Thu, 22 Oct 2015, Barry Smith wrote: > > > >> > >>> On Oct 22, 2015, at 7:28 PM, Satish Balay <[email protected]> wrote: > >>> > >>> The current infrastruture tests --download-petsc4py in petsc 'next' > >>> and 'master' branches. > >>> > >>> Wrt next - usually a change in a feature branch (when it gets merged > >>> to next) triggers this breakage. And usual thing to do is to fix it > >>> in the feature branch (and merge to next again) > >>> > >>> Wrt to petsc4py - this would mean updating petsc4py.py [in petsc] with > >>> the petsc4py repo gitcommit [optionally also tarball ] that > >>> corresponds to the fix. [one way to handle this on petsc4py repo is to > >>> create a feature branch for this fix] > >> > >> Yeah this is nuts, having to maintaining matching branches between petsc > >> and petsc4py and merge them at the same time into next and then master. > >> Clearly no one will ever do this and hence petsc4py will always be out of > >> step from PETSc. I do really believe that incorporating petsc4py into > >> PETSc would just make all these difficulties go away. For example > >> maintaining Tao is a breeze now since it is updated in the same branches > >> as PETSc. > >> > > > > Perhaps I should rephrase the issue in the following way: > > > > We should decide on what level petsc4py and petsc should be synced. > > [and the testsuite should be modified appropriately for that] > > > > Currently its done at next, master branchs. For this to work - the > > above - previously mentioned workflow is required. [and for this > > workflow - a single repo does makes thing easier] > > > > We could say sync should happen only between petsc4py-master & > > petsc-master. [so issues - if any in feature branches/next wont't be > > dealt with - and its ok for the master branches to be out of sync for > > a brief time for the fix to propergate]. > > > > For this - we would have a different workflow [wrt syncing petsc4py > > wrt petsc - and a slightly different test suite - testing only > > master. [We could continue having the curent test suite - but ignore > > errors in next]. And e-mails to Lisandro will trigger only on > > petsc4py errors on master. Lisandro would then fix petsc4py (master) > > - and then update petsc4py.py in petsc-master. > > > > Or perhaps some other mode is preferable.. > > > > Satish > > > >> Barry > >> > >>> > >>> When this (petsc) feature branch is merged into master - the > >>> corresponding feature branch on the petsc4py side can be merged into > >>> petsc4py master [if petsc4py-master is to be in sync with > >>> petsc-master]. > >>> > >>> [if there are multiple feature branches on the petsc side requiring > >>> corresponding petsc4py changes - somehow that has to be handled] > >>> > >>> Perhaps testing petsc4py via petsc infrastructure [using > >>> --download-petsc4py] is not the appropriate approach? > >>> > >>> Or we should do this only on master - not next? [or just have a > >>> single --download-petsc4py test - thats not mixed with other tests? - > >>> so its breakage is not critical for other things] > >>> > >>> Also we don't currently run any petsc4py tests. [we just test if it > >>> compiles or not] > >>> > >>> Satish > >>> > >>> On Thu, 22 Oct 2015, Barry Smith wrote: > >>> > >>>> > >>>> Lisandro, > >>>> > >>>> It is tested nightly but there is apparently currently no email sent on > >>>> failure. See the bottom of, for example, > >>>> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2015/10/22/make_next_arch-linux-pkgs-opt_crank.log > >>>> > >>>> I think what should happen is the petsc4py log file should be published > >>>> to the web (like the other log files) and a script look through it and > >>>> decide to send email if it detects a problem. Like we currently have for > >>>> the other log files. Someone has to dig into the nightly test > >>>> infrastructure to figure out where the new logic needs to go. Currently > >>>> Satish and Karl understand that infrastructure the best. > >>>> > >>>> Barry > >>>> > >>>> > >>>> > >>>> > >>>> > >>>>> On Oct 22, 2015, at 6:44 PM, Lisandro Dalcin <[email protected]> wrote: > >>>>> > >>>>> On 22 October 2015 at 23:20, Barry Smith <[email protected]> wrote: > >>>>>> > >>>>>> Lisandro, > >>>>>> > >>>>>> You never responded to this. I assume it was because you did not like > >>>>>> the idea? This is not an attempt to take control of petsc4py or to > >>>>>> take credit away from petsc4py from you. Petsc4py is your package and > >>>>>> you will always be the one who deserves the credit. > >>>>>> > >>>>> > >>>>> Oh, sorry! I missed this email. > >>>>> > >>>>> > >>>>>> The problem is that mutual efficient development of very intertwined > >>>>>> packages is difficult if they are in different respositories. For > >>>>>> example petsc4py has been out of date with petsc master for many weeks > >>>>>> now because it is too much effort to update the petsc4py repository > >>>>>> each time we make some change in PETSc. Thus it becomes a burden on > >>>>>> you to go in every once in a while and fix up petsc4py. With one > >>>>>> repository changes would happen in both PETSc source and petsc4py > >>>>>> source at the same time and tests would detect problems immediately. > >>>>>> > >>>>>> Thoughts? > >>>>>> > >>>>> > >>>>> I general terms, I prefer to keep things separate as much a possible. > >>>>> IMHO, a much important step would be to incorporate petsc4py to the > >>>>> nightly tests, clearly separating the petsc and petsc4py tests runs. I > >>>>> would love to get an email every time petsc4py nightly breaks, this > >>>>> way I would be able to push fixes within a day or two, say. > >>>>> Maintaining petsc4py requires knowledge of PETSc+Python+Cython, so > >>>>> ultimately I'm the one in best position to do such work, if you guys > >>>>> start adding commits to petsc4py withing the main petsc repo, I'm > >>>>> likely going to miss these changes. For example, today Michael Lange > >>>>> made some changes, I got an email from Bitbucket, then did a review > >>>>> and spotted a few issues. > >>>>> > >>>>> My hesitation to incorporate petsc4py within petsc repo is, in > >>>>> Python's [and linear algebra :-)] Zen : "Sparse is better than dense". > >>>>> If we continue adding stuff to core PETSc, at same point we will endup > >>>>> like Trilinos ( DON'T MISS the sencond entry in the FAQ: > >>>>> https://trilinos.org/download/public-git-repository/). > >>>>> > >>>>> All that being said, I don't have any problem at all adding petsc4py > >>>>> to the main PETSc repository. But I still think this is not a good > >>>>> idea, it will not solve all the problems. Nightly tests with automatic > >>>>> notifications to me about breakages is perhaps is all what we need to > >>>>> keep petsc4py in close sync with petsc/master. What about trying this > >>>>> first to see how it goes? > >>>>> > >>>>> > >>>>> PS: Please understand my position has nothing to do with credits about > >>>>> the project or ego-related crap. Actually, your proposal would mean > >>>>> less responsibility and maintenance work for me. But I don't think it > >>>>> is a move in the right direction. > >>>>> > >>>>> > >>>>> -- > >>>>> Lisandro Dalcin > >>>>> ============ > >>>>> Research Scientist > >>>>> Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) > >>>>> Numerical Porous Media Center (NumPor) > >>>>> King Abdullah University of Science and Technology (KAUST) > >>>>> http://numpor.kaust.edu.sa/ > >>>>> > >>>>> 4700 King Abdullah University of Science and Technology > >>>>> al-Khawarizmi Bldg (Bldg 1), Office # 4332 > >>>>> Thuwal 23955-6900, Kingdom of Saudi Arabia > >>>>> http://www.kaust.edu.sa > >>>>> > >>>>> Office Phone: +966 12 808-0459 > >
