It seems that a few groups are talking past each other. * A sizable contingent is interested in a move to Gradle -- it shows promise, but the work is incomplete. * Another contingent noticing the large burden of maintaining multiple build systems. FWICT, both test suites have been broken quite a lot recently, mainly the gradle ones, which is a cost to the community. This is creating a barrier to entry for new contributors – especially those who don't work in the same room or do their primary development in a different repository.
I don't see this situation being resolved to anyone's satisfaction until there's only one build system left. The onus is clearly on the Gradle promoters to finish the work. Luke made a suggestion 2.5 months ago that we should have a process vote if this situation is untenable. It seems like we're there. Thanks, Dan On Thu, Mar 22, 2018 at 10:30 AM, Romain Manni-Bucau <[email protected]> wrote: > Ok so to be clear for any contributor (which is the goal of this thread): > maven is still the main build system and no need to maintain gradle in PR > then until beam switches. > > Im more than fine with that. > > Le 22 mars 2018 18:22, "Alan Myrvold" <[email protected]> a écrit : > >> I think the investment in gradle is worthwhile, and incrementally we will >> continue to make progress. From what I've seem, gradle is a good fit for >> this project and a path to a faster, more reliable build system. >> >> pull/4812 <https://github.com/apache/beam/pull/4812> creates the release >> artifacts, although it is not hooked up yet with authentication. >> >> I expect to help make incremental progress over the next month converting >> some of the integration tests, but welcome incremental improvements from >> others. >> >> >> >> On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau <[email protected]> >> wrote: >> >>> >>> >>> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik <[email protected]>: >>> >>>> what do we do? "Gradle migration will happen incrementally." >>>> >>>> "last months prooved beam cant maintain 2 systems, easier with that >>>> state is then to drop gradle since it is a 0 investment compared to the >>>> opposite" >>>> Its unfortunate that you feel this way but many people do not share >>>> your opinion. >>>> >>> >>> And a lot do so when a project is 50-50 it is time to act. >>> >>> Incrementally kind of means never (makes 4 months and nothing really >>> changed in PRs and habits, gradle maintener(s) are still alone) >>> >>> >>>> >>>> >>>> On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau < >>>> [email protected]> wrote: >>>> >>>>> @Valentyn: concretely any user can PR and be part of that process so >>>>> anyone can do it wrong (me first) >>>>> @Luskasz, Hennking: fine but what do we do? last months prooved beam >>>>> cant maintain 2 systems, easier with that state is then to drop gradle >>>>> since it is a 0 investment compared to the opposite >>>>> >>>>> >>>>> Romain Manni-Bucau >>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>> <http://rmannibucau.wordpress.com> | Github >>>>> <https://github.com/rmannibucau> | LinkedIn >>>>> <https://www.linkedin.com/in/rmannibucau> | Book >>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance> >>>>> >>>>> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik <[email protected]>: >>>>> >>>>>> Romain, from the previous discussions several people agreed that >>>>>> running a fixit that migrated Maven to Gradle over a 1 or 2 day period >>>>>> was >>>>>> worthwhile but there was nobody in the community with the time commitment >>>>>> to organize and run it so the status quo plan remained where the Gradle >>>>>> migration will happen incrementally. >>>>>> >>>>>> >>>>>> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> My understanding was the same as Ismaël's. I don't think breaking >>>>>>> the build with a large known gaps (but not fully known cost) is >>>>>>> practical. >>>>>>> Also, most items in the jira are not even assigned yet. >>>>>>> >>>>>>> >>>>>>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Not really Ismaël, this thread was about to do it at once and have >>>>>>>> 1 day to fix it all. >>>>>>>> >>>>>>>> As mentionned at the very beginning nobody maintains the 2 system >>>>>>>> so it must stop after months so either we drop maven or gradle *at >>>>>>>> once* >>>>>>>> or we keep a state where each dev does what he wants and the build >>>>>>>> system just doesn't work. >>>>>>>> >>>>>>>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía <[email protected]>: >>>>>>>> >>>>>>>>> I don't think that removing all maven descriptors was the expected >>>>>>>>> path, no ? Or even a good idea at this moment. >>>>>>>>> >>>>>>>>> I understood that what we were going to do was to replace >>>>>>>>> incrementally the CI until we cover the whole maven functionality >>>>>>>>> and >>>>>>>>> then remove it, from looking at the JIRA ticket >>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far >>>>>>>>> from >>>>>>>>> covering the complete maven functionality in particular for the >>>>>>>>> release part that could be the biggest pain point. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau >>>>>>>>> <[email protected]> wrote: >>>>>>>>> > hey guys, >>>>>>>>> > >>>>>>>>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or >>>>>>>>> on monday? >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > Romain Manni-Bucau >>>>>>>>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>>>>>>> > >>>>>>>>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles <[email protected]>: >>>>>>>>> >> >>>>>>>>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>> >>>>>>>>> >>> Based upon your description it seems as though you would >>>>>>>>> rather have a >>>>>>>>> >>> way to run existing postcommits without it impacting >>>>>>>>> dashboard/health >>>>>>>>> >>> stats/notifications/.... (We have just run the PostCommits on >>>>>>>>> PRs for >>>>>>>>> >>> additional validation (like upgrading the Dataflow container >>>>>>>>> image)). >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> Yes, that is exactly what I have described. >>>>>>>>> >> >>>>>>>>> >>> I don't think that keeping the current Java PreCommit as a >>>>>>>>> proxy for the >>>>>>>>> >>> the Java PostCommit is the right way to go but I also don't >>>>>>>>> have the time to >>>>>>>>> >>> implement what your actually asking for. >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> Mostly I thought this might be very easy based on the fact that >>>>>>>>> they are >>>>>>>>> >> nearly identical. If not, oh well. >>>>>>>>> >> >>>>>>>>> >> Kenn >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >>> It seems more likely that migrating the PostCommit to Gradle >>>>>>>>> will be less >>>>>>>>> >>> work then adding the functionality but your argument where the >>>>>>>>> PreCommit is >>>>>>>>> >>> a proxy for the Java PostCommit also applies to the >>>>>>>>> ValidatesRunner >>>>>>>>> >>> PostCommits and so forth requiring even more migration to >>>>>>>>> happen before you >>>>>>>>> >>> don't have to worry about maintaining Maven/breaking post >>>>>>>>> commits. >>>>>>>>> >>> >>>>>>>>> >>> I'm fine with leaving both the Java/Gradle PreCommits running >>>>>>>>> for now and >>>>>>>>> >>> hopefully as more of the PostCommits are migrated off we will >>>>>>>>> be able to >>>>>>>>> >>> remove it. >>>>>>>>> >>> >>>>>>>>> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>> >>>>>>>>> >>>> Separate history (for easy dashboarding, health stats, etc) >>>>>>>>> and >>>>>>>>> >>>> notification (email to dev@ for postcommits, nothing for >>>>>>>>> precommits) for pre >>>>>>>>> >>>> & post commit targets. >>>>>>>>> >>>> >>>>>>>>> >>>> A post commit failure is always a problem to be triaged at >>>>>>>>> high >>>>>>>>> >>>> priority, while a precommit failure is just a natural >>>>>>>>> occurrence. >>>>>>>>> >>>> >>>>>>>>> >>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>> >>>>>>>>> >>>>> Ken, I'm probably not seeing something but how does using >>>>>>>>> the PreCommit >>>>>>>>> >>>>> as a proxy improve upon just running the post commit via the >>>>>>>>> phrase it >>>>>>>>> >>>>> already supports ('Run Java PostCommit')? >>>>>>>>> >>>>> >>>>>>>>> >>>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles < >>>>>>>>> [email protected]> >>>>>>>>> >>>>> wrote: >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> Indeed, we've already had the discussion a couple of times >>>>>>>>> and I think >>>>>>>>> >>>>>> the criteria are clearly met. Incremental progress is a >>>>>>>>> good thing and we >>>>>>>>> >>>>>> shouldn't block it. >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> OTOH I see where Romain is coming from and I have a good >>>>>>>>> example that >>>>>>>>> >>>>>> supports a slightly different action. Consider >>>>>>>>> >>>>>> https://github.com/apache/beam/pull/4740 which fixes some >>>>>>>>> errors in how we >>>>>>>>> >>>>>> use dependency mechanisms. >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> This PR is green except that I need to fix some Maven pom >>>>>>>>> slightly >>>>>>>>> >>>>>> more. That is throwaway work. I would love to just not have >>>>>>>>> to do it. But >>>>>>>>> >>>>>> removing the precommit does not actually make the PR OK to >>>>>>>>> merge. It would >>>>>>>>> >>>>>> cause postcommits to fail. >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> We can hope such situations are rare. I think I tend to be >>>>>>>>> hit by this >>>>>>>>> >>>>>> more often than most, as I work with the project build >>>>>>>>> health quite a bit. >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> Here is a proposal to support these things: instead of >>>>>>>>> deleting the >>>>>>>>> >>>>>> job in #4814, move it to not run automatically but only via >>>>>>>>> a phrase. In >>>>>>>>> >>>>>> fact, you could migrate it to be the manually-invoked >>>>>>>>> version of the >>>>>>>>> >>>>>> postcommit job as we've discussed a couple times. Then if >>>>>>>>> someone is working >>>>>>>>> >>>>>> on the build in something like #4740 they can invoke it >>>>>>>>> manually. >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> Kenn >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>> >>>>>>>>> >>>>>>> Based upon the criteria that was discussed on the mailing >>>>>>>>> list[1], I >>>>>>>>> >>>>>>> would agree with Kenn about merging PR/4814 (drop Java >>>>>>>>> Maven precommit). >>>>>>>>> >>>>>>> >>>>>>>>> >>>>>>> 1: >>>>>>>>> >>>>>>> https://lists.apache.org/threa >>>>>>>>> d.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5 >>>>>>>>> e99@%3Cdev.beam.apache.org%3E >>>>>>>>> >>>>>>> >>>>>>>>> >>>>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau >>>>>>>>> >>>>>>> <[email protected]> wrote: >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> Hi Kenneth, >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> For now maven covers the full needs of beam. If we start >>>>>>>>> to have >>>>>>>>> >>>>>>>> this kind of PR we become dependent of the 2 builds which >>>>>>>>> is what this >>>>>>>>> >>>>>>>> thread is about avoiding so tempted to say it must be a >>>>>>>>> PR drop completely >>>>>>>>> >>>>>>>> maven or nothing as mentionned before. >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <[email protected]> >>>>>>>>> a écrit : >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I would like to briefly re-focus this discussion and >>>>>>>>> suggest that >>>>>>>>> >>>>>>>>> we merge https://github.com/apache/beam/pull/4814. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> The only material objection I've heard is that it means >>>>>>>>> the >>>>>>>>> >>>>>>>>> precommit no longer tests exactly what is built for >>>>>>>>> release. It is a valid >>>>>>>>> >>>>>>>>> concern, but we have mvn postcommit so the coverage is >>>>>>>>> not lost. The >>>>>>>>> >>>>>>>>> benefits of quicker reviews and less Jenkins congestion >>>>>>>>> seem worth it to me. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Kenn >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau >>>>>>>>> >>>>>>>>> <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> @Luskasz: not sure Im the best to host it since I know >>>>>>>>> more gradle >>>>>>>>> >>>>>>>>>> internals that user interface/ecosystem but happy to >>>>>>>>> help. Will also require >>>>>>>>> >>>>>>>>>> a "sudo" merger for this day to merge fixes asap - >>>>>>>>> guess we can bypass >>>>>>>>> >>>>>>>>>> reviews or have a fast cycle plan for this day to avoid >>>>>>>>> it to be a week? >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <[email protected]> >>>>>>>>> a écrit : >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way >>>>>>>>> better than >>>>>>>>> >>>>>>>>>> Maven, and what I mean is that I have added enough >>>>>>>>> hints that it works OOTB >>>>>>>>> >>>>>>>>>> already. The rest of my instructions are just how you >>>>>>>>> should override >>>>>>>>> >>>>>>>>>> IntelliJ's defaults to have a proper dev env - mostly >>>>>>>>> just about storing >>>>>>>>> >>>>>>>>>> files outside the git clone. >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> Hmm it is always super slow here and not as integrated >>>>>>>>> as maven >>>>>>>>> >>>>>>>>>> which is smooth thanks to the combination of idea build >>>>>>>>> and maven plugin >>>>>>>>> >>>>>>>>>> config read. Import is also faster cause it just reads >>>>>>>>> meta and doesnt run >>>>>>>>> >>>>>>>>>> anything. Hope it is a "a few times" issues at the >>>>>>>>> moment but not yet sure. >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> @Łukasz - yea I just mean the ones that I find in my >>>>>>>>> email and >>>>>>>>> >>>>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/ >>>>>>>>> which are the ones >>>>>>>>> >>>>>>>>>> you classify as "totally not working". I would love to >>>>>>>>> disable these. >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> On the subject of running things on a pending PR - we >>>>>>>>> should not >>>>>>>>> >>>>>>>>>> run postcommit jobs on PRs. We should make separate >>>>>>>>> (optional) precommit >>>>>>>>> >>>>>>>>>> jobs that run the same build commands. That will give a >>>>>>>>> more clear history >>>>>>>>> >>>>>>>>>> and allow trivial configuration of which jobs deserve >>>>>>>>> alert emails and which >>>>>>>>> >>>>>>>>>> are not a problem. This is easy but I've been waiting >>>>>>>>> to do it after Gradle >>>>>>>>> >>>>>>>>>> migration. >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik < >>>>>>>>> [email protected]> >>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>>>> Romain, would you like to host/plan/run the Gradle >>>>>>>>> fixit day? >>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath >>>>>>>>> >>>>>>>>>>> <[email protected]> wrote: >>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>> +1 for the general idea of fixit day/week for Gradle. >>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>> Agree with what Łukasz said. Some of these >>>>>>>>> performance tests are >>>>>>>>> >>>>>>>>>>>> new and are flaky due to other issues that were >>>>>>>>> discovered during the >>>>>>>>> >>>>>>>>>>>> process of adding the test. >>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>> I think the high level blocker is updating >>>>>>>>> performance testing >>>>>>>>> >>>>>>>>>>>> framework to use Gradle. This will involve adding >>>>>>>>> Gradle-based logic for >>>>>>>>> >>>>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and >>>>>>>>> updating PerfKitBenchmarker >>>>>>>>> >>>>>>>>>>>> to issue a Gradle command for running the benchmark >>>>>>>>> [2]. First task will be >>>>>>>>> >>>>>>>>>>>> to find the work needed here and updating the >>>>>>>>> relevant JIRA [3]. >>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>>>>> Cham >>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>> [1] >>>>>>>>> >>>>>>>>>>>> https://github.com/apache/beam >>>>>>>>> /blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79 >>>>>>>>> >>>>>>>>>>>> [2] >>>>>>>>> >>>>>>>>>>>> https://github.com/GoogleCloud >>>>>>>>> Platform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_ >>>>>>>>> benchmark_helper.py >>>>>>>>> >>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251 >>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy >>>>>>>>> >>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles < >>>>>>>>> [email protected]>: >>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> Based on https://builds.apache.org/view >>>>>>>>> /A-D/view/Beam/ and our >>>>>>>>> >>>>>>>>>>>>>> failure spam level the performance tests are mostly >>>>>>>>> not healthy anyhow. So >>>>>>>>> >>>>>>>>>>>>>> is there any high level blocker to switching them >>>>>>>>> or is it just someone >>>>>>>>> >>>>>>>>>>>>>> sitting down with each one? >>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>> I thought I could share my point of view here as I >>>>>>>>> am working >>>>>>>>> >>>>>>>>>>>>> on the Performance Test part for a while now. I >>>>>>>>> wouldn't say those are >>>>>>>>> >>>>>>>>>>>>> "mostly not healthy". The situation is as follows: >>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>> - totally not working: Python, Spark performance >>>>>>>>> tests (don't >>>>>>>>> >>>>>>>>>>>>> know why, I'm not maintaining those tests. Should we >>>>>>>>> disable them?) >>>>>>>>> >>>>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance >>>>>>>>> Test. It's >>>>>>>>> >>>>>>>>>>>>> seems to be flaky mostly due to: >>>>>>>>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798 >>>>>>>>> >>>>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO, >>>>>>>>> Compressed >>>>>>>>> >>>>>>>>>>>>> Text, TFRecordIO >>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>> Also, some test failures are caused due to pending >>>>>>>>> PRs (we >>>>>>>>> >>>>>>>>>>>>> sometimes run concrete tests to check if PRs won't >>>>>>>>> break them). This also >>>>>>>>> >>>>>>>>>>>>> causes failures sometimes. >>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>> I can help with switching the Performance tests to >>>>>>>>> Gradle as >>>>>>>>> >>>>>>>>>>>>> this part seems to be free to take. >>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>> Łukasz >>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>>>> >>>>> >>>>>>>>> >>> >>>>>>>>> > >>>>>>>>> >>>>>>>> >>>>>>>> >>>>> >>>
