Le 22 mars 2018 18:49, "Dan Halperin" <dhalp...@apache.org> a écrit :
It seems that a few groups are talking past each other. * A sizable contingent is interested in a move to Gradle -- it shows promise, but the work is incomplete. * Another contingent noticing the large burden of maintaining multiple build systems. FWICT, both test suites have been broken quite a lot recently, mainly the gradle ones, which is a cost to the community. This is creating a barrier to entry for new contributors – especially those who don't work in the same room or do their primary development in a different repository. I don't see this situation being resolved to anyone's satisfaction until there's only one build system left. The onus is clearly on the Gradle promoters to finish the work. Luke made a suggestion 2.5 months ago that we should have a process vote if this situation is untenable. It seems like we're there. Yes but beam voted to move to gradle so we should but we shouldnt maintain 2 build systems for more than 2 months (weway overpassed that) and therefore the vote should be cancelled or validated by an action. I understand you want gradle but you dont want to pay the cost to move to gradle, it doesnt work for the project do please another option (rollbacking gradle or removing maven, all other options are negative for the project and a pain for committers and contributors whatever you think). Thanks, Dan On Thu, Mar 22, 2018 at 10:30 AM, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > Ok so to be clear for any contributor (which is the goal of this thread): > maven is still the main build system and no need to maintain gradle in PR > then until beam switches. > > Im more than fine with that. > > Le 22 mars 2018 18:22, "Alan Myrvold" <amyrv...@google.com> a écrit : > >> I think the investment in gradle is worthwhile, and incrementally we will >> continue to make progress. From what I've seem, gradle is a good fit for >> this project and a path to a faster, more reliable build system. >> >> pull/4812 <https://github.com/apache/beam/pull/4812> creates the release >> artifacts, although it is not hooked up yet with authentication. >> >> I expect to help make incremental progress over the next month converting >> some of the integration tests, but welcome incremental improvements from >> others. >> >> >> >> On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau <rmannibu...@gmail.com> >> wrote: >> >>> >>> >>> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik <lc...@google.com>: >>> >>>> what do we do? "Gradle migration will happen incrementally." >>>> >>>> "last months prooved beam cant maintain 2 systems, easier with that >>>> state is then to drop gradle since it is a 0 investment compared to the >>>> opposite" >>>> Its unfortunate that you feel this way but many people do not share >>>> your opinion. >>>> >>> >>> And a lot do so when a project is 50-50 it is time to act. >>> >>> Incrementally kind of means never (makes 4 months and nothing really >>> changed in PRs and habits, gradle maintener(s) are still alone) >>> >>> >>>> >>>> >>>> On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau < >>>> rmannibu...@gmail.com> wrote: >>>> >>>>> @Valentyn: concretely any user can PR and be part of that process so >>>>> anyone can do it wrong (me first) >>>>> @Luskasz, Hennking: fine but what do we do? last months prooved beam >>>>> cant maintain 2 systems, easier with that state is then to drop gradle >>>>> since it is a 0 investment compared to the opposite >>>>> >>>>> >>>>> Romain Manni-Bucau >>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>> <http://rmannibucau.wordpress.com> | Github >>>>> <https://github.com/rmannibucau> | LinkedIn >>>>> <https://www.linkedin.com/in/rmannibucau> | Book >>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance> >>>>> >>>>> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik <lc...@google.com>: >>>>> >>>>>> Romain, from the previous discussions several people agreed that >>>>>> running a fixit that migrated Maven to Gradle over a 1 or 2 day period >>>>>> was >>>>>> worthwhile but there was nobody in the community with the time commitment >>>>>> to organize and run it so the status quo plan remained where the Gradle >>>>>> migration will happen incrementally. >>>>>> >>>>>> >>>>>> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde <hero...@google.com> >>>>>> wrote: >>>>>> >>>>>>> My understanding was the same as Ismaël's. I don't think breaking >>>>>>> the build with a large known gaps (but not fully known cost) is >>>>>>> practical. >>>>>>> Also, most items in the jira are not even assigned yet. >>>>>>> >>>>>>> >>>>>>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau < >>>>>>> rmannibu...@gmail.com> wrote: >>>>>>> >>>>>>>> Not really Ismaël, this thread was about to do it at once and have >>>>>>>> 1 day to fix it all. >>>>>>>> >>>>>>>> As mentionned at the very beginning nobody maintains the 2 system >>>>>>>> so it must stop after months so either we drop maven or gradle *at >>>>>>>> once* >>>>>>>> or we keep a state where each dev does what he wants and the build >>>>>>>> system just doesn't work. >>>>>>>> >>>>>>>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía <ieme...@gmail.com>: >>>>>>>> >>>>>>>>> I don't think that removing all maven descriptors was the expected >>>>>>>>> path, no ? Or even a good idea at this moment. >>>>>>>>> >>>>>>>>> I understood that what we were going to do was to replace >>>>>>>>> incrementally the CI until we cover the whole maven functionality >>>>>>>>> and >>>>>>>>> then remove it, from looking at the JIRA ticket >>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far >>>>>>>>> from >>>>>>>>> covering the complete maven functionality in particular for the >>>>>>>>> release part that could be the biggest pain point. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau >>>>>>>>> <rmannibu...@gmail.com> wrote: >>>>>>>>> > hey guys, >>>>>>>>> > >>>>>>>>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or >>>>>>>>> on monday? >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > Romain Manni-Bucau >>>>>>>>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>>>>>>> > >>>>>>>>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles <k...@google.com>: >>>>>>>>> >> >>>>>>>>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik <lc...@google.com> >>>>>>>>> wrote: >>>>>>>>> >>> >>>>>>>>> >>> Based upon your description it seems as though you would >>>>>>>>> rather have a >>>>>>>>> >>> way to run existing postcommits without it impacting >>>>>>>>> dashboard/health >>>>>>>>> >>> stats/notifications/.... (We have just run the PostCommits on >>>>>>>>> PRs for >>>>>>>>> >>> additional validation (like upgrading the Dataflow container >>>>>>>>> image)). >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> Yes, that is exactly what I have described. >>>>>>>>> >> >>>>>>>>> >>> I don't think that keeping the current Java PreCommit as a >>>>>>>>> proxy for the >>>>>>>>> >>> the Java PostCommit is the right way to go but I also don't >>>>>>>>> have the time to >>>>>>>>> >>> implement what your actually asking for. >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> Mostly I thought this might be very easy based on the fact that >>>>>>>>> they are >>>>>>>>> >> nearly identical. If not, oh well. >>>>>>>>> >> >>>>>>>>> >> Kenn >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >>> It seems more likely that migrating the PostCommit to Gradle >>>>>>>>> will be less >>>>>>>>> >>> work then adding the functionality but your argument where the >>>>>>>>> PreCommit is >>>>>>>>> >>> a proxy for the Java PostCommit also applies to the >>>>>>>>> ValidatesRunner >>>>>>>>> >>> PostCommits and so forth requiring even more migration to >>>>>>>>> happen before you >>>>>>>>> >>> don't have to worry about maintaining Maven/breaking post >>>>>>>>> commits. >>>>>>>>> >>> >>>>>>>>> >>> I'm fine with leaving both the Java/Gradle PreCommits running >>>>>>>>> for now and >>>>>>>>> >>> hopefully as more of the PostCommits are migrated off we will >>>>>>>>> be able to >>>>>>>>> >>> remove it. >>>>>>>>> >>> >>>>>>>>> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles < >>>>>>>>> k...@google.com> wrote: >>>>>>>>> >>>> >>>>>>>>> >>>> Separate history (for easy dashboarding, health stats, etc) >>>>>>>>> and >>>>>>>>> >>>> notification (email to dev@ for postcommits, nothing for >>>>>>>>> precommits) for pre >>>>>>>>> >>>> & post commit targets. >>>>>>>>> >>>> >>>>>>>>> >>>> A post commit failure is always a problem to be triaged at >>>>>>>>> high >>>>>>>>> >>>> priority, while a precommit failure is just a natural >>>>>>>>> occurrence. >>>>>>>>> >>>> >>>>>>>>> >>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <lc...@google.com> >>>>>>>>> wrote: >>>>>>>>> >>>>> >>>>>>>>> >>>>> Ken, I'm probably not seeing something but how does using >>>>>>>>> the PreCommit >>>>>>>>> >>>>> as a proxy improve upon just running the post commit via the >>>>>>>>> phrase it >>>>>>>>> >>>>> already supports ('Run Java PostCommit')? >>>>>>>>> >>>>> >>>>>>>>> >>>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles < >>>>>>>>> k...@google.com> >>>>>>>>> >>>>> wrote: >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> Indeed, we've already had the discussion a couple of times >>>>>>>>> and I think >>>>>>>>> >>>>>> the criteria are clearly met. Incremental progress is a >>>>>>>>> good thing and we >>>>>>>>> >>>>>> shouldn't block it. >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> OTOH I see where Romain is coming from and I have a good >>>>>>>>> example that >>>>>>>>> >>>>>> supports a slightly different action. Consider >>>>>>>>> >>>>>> https://github.com/apache/beam/pull/4740 which fixes some >>>>>>>>> errors in how we >>>>>>>>> >>>>>> use dependency mechanisms. >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> This PR is green except that I need to fix some Maven pom >>>>>>>>> slightly >>>>>>>>> >>>>>> more. That is throwaway work. I would love to just not have >>>>>>>>> to do it. But >>>>>>>>> >>>>>> removing the precommit does not actually make the PR OK to >>>>>>>>> merge. It would >>>>>>>>> >>>>>> cause postcommits to fail. >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> We can hope such situations are rare. I think I tend to be >>>>>>>>> hit by this >>>>>>>>> >>>>>> more often than most, as I work with the project build >>>>>>>>> health quite a bit. >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> Here is a proposal to support these things: instead of >>>>>>>>> deleting the >>>>>>>>> >>>>>> job in #4814, move it to not run automatically but only via >>>>>>>>> a phrase. In >>>>>>>>> >>>>>> fact, you could migrate it to be the manually-invoked >>>>>>>>> version of the >>>>>>>>> >>>>>> postcommit job as we've discussed a couple times. Then if >>>>>>>>> someone is working >>>>>>>>> >>>>>> on the build in something like #4740 they can invoke it >>>>>>>>> manually. >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> Kenn >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik < >>>>>>>>> lc...@google.com> wrote: >>>>>>>>> >>>>>>> >>>>>>>>> >>>>>>> Based upon the criteria that was discussed on the mailing >>>>>>>>> list[1], I >>>>>>>>> >>>>>>> would agree with Kenn about merging PR/4814 (drop Java >>>>>>>>> Maven precommit). >>>>>>>>> >>>>>>> >>>>>>>>> >>>>>>> 1: >>>>>>>>> >>>>>>> https://lists.apache.org/threa >>>>>>>>> d.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5 >>>>>>>>> e99@%3Cdev.beam.apache.org%3E >>>>>>>>> >>>>>>> >>>>>>>>> >>>>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau >>>>>>>>> >>>>>>> <rmannibu...@gmail.com> wrote: >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> Hi Kenneth, >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> For now maven covers the full needs of beam. If we start >>>>>>>>> to have >>>>>>>>> >>>>>>>> this kind of PR we become dependent of the 2 builds which >>>>>>>>> is what this >>>>>>>>> >>>>>>>> thread is about avoiding so tempted to say it must be a >>>>>>>>> PR drop completely >>>>>>>>> >>>>>>>> maven or nothing as mentionned before. >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <k...@google.com> >>>>>>>>> a écrit : >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I would like to briefly re-focus this discussion and >>>>>>>>> suggest that >>>>>>>>> >>>>>>>>> we merge https://github.com/apache/beam/pull/4814. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> The only material objection I've heard is that it means >>>>>>>>> the >>>>>>>>> >>>>>>>>> precommit no longer tests exactly what is built for >>>>>>>>> release. It is a valid >>>>>>>>> >>>>>>>>> concern, but we have mvn postcommit so the coverage is >>>>>>>>> not lost. The >>>>>>>>> >>>>>>>>> benefits of quicker reviews and less Jenkins congestion >>>>>>>>> seem worth it to me. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Kenn >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau >>>>>>>>> >>>>>>>>> <rmannibu...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> @Luskasz: not sure Im the best to host it since I know >>>>>>>>> more gradle >>>>>>>>> >>>>>>>>>> internals that user interface/ecosystem but happy to >>>>>>>>> help. Will also require >>>>>>>>> >>>>>>>>>> a "sudo" merger for this day to merge fixes asap - >>>>>>>>> guess we can bypass >>>>>>>>> >>>>>>>>>> reviews or have a fast cycle plan for this day to avoid >>>>>>>>> it to be a week? >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <k...@google.com> >>>>>>>>> a écrit : >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way >>>>>>>>> better than >>>>>>>>> >>>>>>>>>> Maven, and what I mean is that I have added enough >>>>>>>>> hints that it works OOTB >>>>>>>>> >>>>>>>>>> already. The rest of my instructions are just how you >>>>>>>>> should override >>>>>>>>> >>>>>>>>>> IntelliJ's defaults to have a proper dev env - mostly >>>>>>>>> just about storing >>>>>>>>> >>>>>>>>>> files outside the git clone. >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> Hmm it is always super slow here and not as integrated >>>>>>>>> as maven >>>>>>>>> >>>>>>>>>> which is smooth thanks to the combination of idea build >>>>>>>>> and maven plugin >>>>>>>>> >>>>>>>>>> config read. Import is also faster cause it just reads >>>>>>>>> meta and doesnt run >>>>>>>>> >>>>>>>>>> anything. Hope it is a "a few times" issues at the >>>>>>>>> moment but not yet sure. >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> @Łukasz - yea I just mean the ones that I find in my >>>>>>>>> email and >>>>>>>>> >>>>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/ >>>>>>>>> which are the ones >>>>>>>>> >>>>>>>>>> you classify as "totally not working". I would love to >>>>>>>>> disable these. >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> On the subject of running things on a pending PR - we >>>>>>>>> should not >>>>>>>>> >>>>>>>>>> run postcommit jobs on PRs. We should make separate >>>>>>>>> (optional) precommit >>>>>>>>> >>>>>>>>>> jobs that run the same build commands. That will give a >>>>>>>>> more clear history >>>>>>>>> >>>>>>>>>> and allow trivial configuration of which jobs deserve >>>>>>>>> alert emails and which >>>>>>>>> >>>>>>>>>> are not a problem. This is easy but I've been waiting >>>>>>>>> to do it after Gradle >>>>>>>>> >>>>>>>>>> migration. >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik < >>>>>>>>> lc...@google.com> >>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>>>> Romain, would you like to host/plan/run the Gradle >>>>>>>>> fixit day? >>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath >>>>>>>>> >>>>>>>>>>> <chamik...@google.com> wrote: >>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>> +1 for the general idea of fixit day/week for Gradle. >>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>> Agree with what Łukasz said. Some of these >>>>>>>>> performance tests are >>>>>>>>> >>>>>>>>>>>> new and are flaky due to other issues that were >>>>>>>>> discovered during the >>>>>>>>> >>>>>>>>>>>> process of adding the test. >>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>> I think the high level blocker is updating >>>>>>>>> performance testing >>>>>>>>> >>>>>>>>>>>> framework to use Gradle. This will involve adding >>>>>>>>> Gradle-based logic for >>>>>>>>> >>>>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and >>>>>>>>> updating PerfKitBenchmarker >>>>>>>>> >>>>>>>>>>>> to issue a Gradle command for running the benchmark >>>>>>>>> [2]. First task will be >>>>>>>>> >>>>>>>>>>>> to find the work needed here and updating the >>>>>>>>> relevant JIRA [3]. >>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>>>>> Cham >>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>> [1] >>>>>>>>> >>>>>>>>>>>> https://github.com/apache/beam >>>>>>>>> /blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79 >>>>>>>>> >>>>>>>>>>>> [2] >>>>>>>>> >>>>>>>>>>>> https://github.com/GoogleCloud >>>>>>>>> Platform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/ >>>>>>>>> beam_benchmark_helper.py >>>>>>>>> >>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251 >>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy >>>>>>>>> >>>>>>>>>>>> <lukasz.gaj...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles < >>>>>>>>> k...@google.com>: >>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> Based on https://builds.apache.org/view >>>>>>>>> /A-D/view/Beam/ and our >>>>>>>>> >>>>>>>>>>>>>> failure spam level the performance tests are mostly >>>>>>>>> not healthy anyhow. So >>>>>>>>> >>>>>>>>>>>>>> is there any high level blocker to switching them >>>>>>>>> or is it just someone >>>>>>>>> >>>>>>>>>>>>>> sitting down with each one? >>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>> I thought I could share my point of view here as I >>>>>>>>> am working >>>>>>>>> >>>>>>>>>>>>> on the Performance Test part for a while now. I >>>>>>>>> wouldn't say those are >>>>>>>>> >>>>>>>>>>>>> "mostly not healthy". The situation is as follows: >>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>> - totally not working: Python, Spark performance >>>>>>>>> tests (don't >>>>>>>>> >>>>>>>>>>>>> know why, I'm not maintaining those tests. Should we >>>>>>>>> disable them?) >>>>>>>>> >>>>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance >>>>>>>>> Test. It's >>>>>>>>> >>>>>>>>>>>>> seems to be flaky mostly due to: >>>>>>>>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798 >>>>>>>>> >>>>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO, >>>>>>>>> Compressed >>>>>>>>> >>>>>>>>>>>>> Text, TFRecordIO >>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>> Also, some test failures are caused due to pending >>>>>>>>> PRs (we >>>>>>>>> >>>>>>>>>>>>> sometimes run concrete tests to check if PRs won't >>>>>>>>> break them). This also >>>>>>>>> >>>>>>>>>>>>> causes failures sometimes. >>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>> I can help with switching the Performance tests to >>>>>>>>> Gradle as >>>>>>>>> >>>>>>>>>>>>> this part seems to be free to take. >>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>> Łukasz >>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>>>> >>>>> >>>>>>>>> >>> >>>>>>>>> > >>>>>>>>> >>>>>>>> >>>>>>>> >>>>> >>>
