what do we do? "Gradle migration will happen incrementally." "last months prooved beam cant maintain 2 systems, easier with that state is then to drop gradle since it is a 0 investment compared to the opposite" Its unfortunate that you feel this way but many people do not share your opinion.
On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau <[email protected]> wrote: > @Valentyn: concretely any user can PR and be part of that process so > anyone can do it wrong (me first) > @Luskasz, Hennking: fine but what do we do? last months prooved beam cant > maintain 2 systems, easier with that state is then to drop gradle since it > is a 0 investment compared to the opposite > > > Romain Manni-Bucau > @rmannibucau <https://twitter.com/rmannibucau> | Blog > <https://rmannibucau.metawerx.net/> | Old Blog > <http://rmannibucau.wordpress.com> | Github > <https://github.com/rmannibucau> | LinkedIn > <https://www.linkedin.com/in/rmannibucau> | Book > <https://www.packtpub.com/application-development/java-ee-8-high-performance> > > 2018-03-22 17:24 GMT+01:00 Lukasz Cwik <[email protected]>: > >> Romain, from the previous discussions several people agreed that running >> a fixit that migrated Maven to Gradle over a 1 or 2 day period was >> worthwhile but there was nobody in the community with the time commitment >> to organize and run it so the status quo plan remained where the Gradle >> migration will happen incrementally. >> >> >> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde <[email protected]> wrote: >> >>> My understanding was the same as Ismaël's. I don't think breaking the >>> build with a large known gaps (but not fully known cost) is practical. >>> Also, most items in the jira are not even assigned yet. >>> >>> >>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau < >>> [email protected]> wrote: >>> >>>> Not really Ismaël, this thread was about to do it at once and have 1 >>>> day to fix it all. >>>> >>>> As mentionned at the very beginning nobody maintains the 2 system so it >>>> must stop after months so either we drop maven or gradle *at once* >>>> or we keep a state where each dev does what he wants and the build >>>> system just doesn't work. >>>> >>>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía <[email protected]>: >>>> >>>>> I don't think that removing all maven descriptors was the expected >>>>> path, no ? Or even a good idea at this moment. >>>>> >>>>> I understood that what we were going to do was to replace >>>>> incrementally the CI until we cover the whole maven functionality and >>>>> then remove it, from looking at the JIRA ticket >>>>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far from >>>>> covering the complete maven functionality in particular for the >>>>> release part that could be the biggest pain point. >>>>> >>>>> >>>>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau >>>>> <[email protected]> wrote: >>>>> > hey guys, >>>>> > >>>>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on >>>>> monday? >>>>> > >>>>> > >>>>> > Romain Manni-Bucau >>>>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>>> > >>>>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles <[email protected]>: >>>>> >> >>>>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik <[email protected]> >>>>> wrote: >>>>> >>> >>>>> >>> Based upon your description it seems as though you would rather >>>>> have a >>>>> >>> way to run existing postcommits without it impacting >>>>> dashboard/health >>>>> >>> stats/notifications/.... (We have just run the PostCommits on PRs >>>>> for >>>>> >>> additional validation (like upgrading the Dataflow container >>>>> image)). >>>>> >> >>>>> >> >>>>> >> Yes, that is exactly what I have described. >>>>> >> >>>>> >>> I don't think that keeping the current Java PreCommit as a proxy >>>>> for the >>>>> >>> the Java PostCommit is the right way to go but I also don't have >>>>> the time to >>>>> >>> implement what your actually asking for. >>>>> >> >>>>> >> >>>>> >> Mostly I thought this might be very easy based on the fact that >>>>> they are >>>>> >> nearly identical. If not, oh well. >>>>> >> >>>>> >> Kenn >>>>> >> >>>>> >> >>>>> >>> It seems more likely that migrating the PostCommit to Gradle will >>>>> be less >>>>> >>> work then adding the functionality but your argument where the >>>>> PreCommit is >>>>> >>> a proxy for the Java PostCommit also applies to the ValidatesRunner >>>>> >>> PostCommits and so forth requiring even more migration to happen >>>>> before you >>>>> >>> don't have to worry about maintaining Maven/breaking post commits. >>>>> >>> >>>>> >>> I'm fine with leaving both the Java/Gradle PreCommits running for >>>>> now and >>>>> >>> hopefully as more of the PostCommits are migrated off we will be >>>>> able to >>>>> >>> remove it. >>>>> >>> >>>>> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles <[email protected]> >>>>> wrote: >>>>> >>>> >>>>> >>>> Separate history (for easy dashboarding, health stats, etc) and >>>>> >>>> notification (email to dev@ for postcommits, nothing for >>>>> precommits) for pre >>>>> >>>> & post commit targets. >>>>> >>>> >>>>> >>>> A post commit failure is always a problem to be triaged at high >>>>> >>>> priority, while a precommit failure is just a natural occurrence. >>>>> >>>> >>>>> >>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <[email protected]> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> Ken, I'm probably not seeing something but how does using the >>>>> PreCommit >>>>> >>>>> as a proxy improve upon just running the post commit via the >>>>> phrase it >>>>> >>>>> already supports ('Run Java PostCommit')? >>>>> >>>>> >>>>> >>>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles <[email protected] >>>>> > >>>>> >>>>> wrote: >>>>> >>>>>> >>>>> >>>>>> Indeed, we've already had the discussion a couple of times and >>>>> I think >>>>> >>>>>> the criteria are clearly met. Incremental progress is a good >>>>> thing and we >>>>> >>>>>> shouldn't block it. >>>>> >>>>>> >>>>> >>>>>> OTOH I see where Romain is coming from and I have a good >>>>> example that >>>>> >>>>>> supports a slightly different action. Consider >>>>> >>>>>> https://github.com/apache/beam/pull/4740 which fixes some >>>>> errors in how we >>>>> >>>>>> use dependency mechanisms. >>>>> >>>>>> >>>>> >>>>>> This PR is green except that I need to fix some Maven pom >>>>> slightly >>>>> >>>>>> more. That is throwaway work. I would love to just not have to >>>>> do it. But >>>>> >>>>>> removing the precommit does not actually make the PR OK to >>>>> merge. It would >>>>> >>>>>> cause postcommits to fail. >>>>> >>>>>> >>>>> >>>>>> We can hope such situations are rare. I think I tend to be hit >>>>> by this >>>>> >>>>>> more often than most, as I work with the project build health >>>>> quite a bit. >>>>> >>>>>> >>>>> >>>>>> Here is a proposal to support these things: instead of deleting >>>>> the >>>>> >>>>>> job in #4814, move it to not run automatically but only via a >>>>> phrase. In >>>>> >>>>>> fact, you could migrate it to be the manually-invoked version >>>>> of the >>>>> >>>>>> postcommit job as we've discussed a couple times. Then if >>>>> someone is working >>>>> >>>>>> on the build in something like #4740 they can invoke it >>>>> manually. >>>>> >>>>>> >>>>> >>>>>> Kenn >>>>> >>>>>> >>>>> >>>>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <[email protected]> >>>>> wrote: >>>>> >>>>>>> >>>>> >>>>>>> Based upon the criteria that was discussed on the mailing >>>>> list[1], I >>>>> >>>>>>> would agree with Kenn about merging PR/4814 (drop Java Maven >>>>> precommit). >>>>> >>>>>>> >>>>> >>>>>>> 1: >>>>> >>>>>>> >>>>> https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E >>>>> >>>>>>> >>>>> >>>>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau >>>>> >>>>>>> <[email protected]> wrote: >>>>> >>>>>>>> >>>>> >>>>>>>> Hi Kenneth, >>>>> >>>>>>>> >>>>> >>>>>>>> For now maven covers the full needs of beam. If we start to >>>>> have >>>>> >>>>>>>> this kind of PR we become dependent of the 2 builds which is >>>>> what this >>>>> >>>>>>>> thread is about avoiding so tempted to say it must be a PR >>>>> drop completely >>>>> >>>>>>>> maven or nothing as mentionned before. >>>>> >>>>>>>> >>>>> >>>>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <[email protected]> a >>>>> écrit : >>>>> >>>>>>>>> >>>>> >>>>>>>>> I would like to briefly re-focus this discussion and suggest >>>>> that >>>>> >>>>>>>>> we merge https://github.com/apache/beam/pull/4814. >>>>> >>>>>>>>> >>>>> >>>>>>>>> The only material objection I've heard is that it means the >>>>> >>>>>>>>> precommit no longer tests exactly what is built for release. >>>>> It is a valid >>>>> >>>>>>>>> concern, but we have mvn postcommit so the coverage is not >>>>> lost. The >>>>> >>>>>>>>> benefits of quicker reviews and less Jenkins congestion seem >>>>> worth it to me. >>>>> >>>>>>>>> >>>>> >>>>>>>>> Kenn >>>>> >>>>>>>>> >>>>> >>>>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau >>>>> >>>>>>>>> <[email protected]> wrote: >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> @Luskasz: not sure Im the best to host it since I know more >>>>> gradle >>>>> >>>>>>>>>> internals that user interface/ecosystem but happy to help. >>>>> Will also require >>>>> >>>>>>>>>> a "sudo" merger for this day to merge fixes asap - guess we >>>>> can bypass >>>>> >>>>>>>>>> reviews or have a fast cycle plan for this day to avoid it >>>>> to be a week? >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <[email protected]> a >>>>> écrit : >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way better >>>>> than >>>>> >>>>>>>>>> Maven, and what I mean is that I have added enough hints >>>>> that it works OOTB >>>>> >>>>>>>>>> already. The rest of my instructions are just how you >>>>> should override >>>>> >>>>>>>>>> IntelliJ's defaults to have a proper dev env - mostly just >>>>> about storing >>>>> >>>>>>>>>> files outside the git clone. >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> Hmm it is always super slow here and not as integrated as >>>>> maven >>>>> >>>>>>>>>> which is smooth thanks to the combination of idea build and >>>>> maven plugin >>>>> >>>>>>>>>> config read. Import is also faster cause it just reads meta >>>>> and doesnt run >>>>> >>>>>>>>>> anything. Hope it is a "a few times" issues at the moment >>>>> but not yet sure. >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> @Łukasz - yea I just mean the ones that I find in my email >>>>> and >>>>> >>>>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/ >>>>> which are the ones >>>>> >>>>>>>>>> you classify as "totally not working". I would love to >>>>> disable these. >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> On the subject of running things on a pending PR - we >>>>> should not >>>>> >>>>>>>>>> run postcommit jobs on PRs. We should make separate >>>>> (optional) precommit >>>>> >>>>>>>>>> jobs that run the same build commands. That will give a >>>>> more clear history >>>>> >>>>>>>>>> and allow trivial configuration of which jobs deserve alert >>>>> emails and which >>>>> >>>>>>>>>> are not a problem. This is easy but I've been waiting to do >>>>> it after Gradle >>>>> >>>>>>>>>> migration. >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik < >>>>> [email protected]> >>>>> >>>>>>>>>> wrote: >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> Romain, would you like to host/plan/run the Gradle fixit >>>>> day? >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath >>>>> >>>>>>>>>>> <[email protected]> wrote: >>>>> >>>>>>>>>>>> >>>>> >>>>>>>>>>>> +1 for the general idea of fixit day/week for Gradle. >>>>> >>>>>>>>>>>> >>>>> >>>>>>>>>>>> Agree with what Łukasz said. Some of these performance >>>>> tests are >>>>> >>>>>>>>>>>> new and are flaky due to other issues that were >>>>> discovered during the >>>>> >>>>>>>>>>>> process of adding the test. >>>>> >>>>>>>>>>>> >>>>> >>>>>>>>>>>> I think the high level blocker is updating performance >>>>> testing >>>>> >>>>>>>>>>>> framework to use Gradle. This will involve adding >>>>> Gradle-based logic for >>>>> >>>>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and updating >>>>> PerfKitBenchmarker >>>>> >>>>>>>>>>>> to issue a Gradle command for running the benchmark [2]. >>>>> First task will be >>>>> >>>>>>>>>>>> to find the work needed here and updating the relevant >>>>> JIRA [3]. >>>>> >>>>>>>>>>>> >>>>> >>>>>>>>>>>> Thanks, >>>>> >>>>>>>>>>>> Cham >>>>> >>>>>>>>>>>> >>>>> >>>>>>>>>>>> [1] >>>>> >>>>>>>>>>>> >>>>> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79 >>>>> >>>>>>>>>>>> [2] >>>>> >>>>>>>>>>>> >>>>> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py >>>>> >>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251 >>>>> >>>>>>>>>>>> >>>>> >>>>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy >>>>> >>>>>>>>>>>> <[email protected]> wrote: >>>>> >>>>>>>>>>>>> >>>>> >>>>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles < >>>>> [email protected]>: >>>>> >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/ >>>>> and our >>>>> >>>>>>>>>>>>>> failure spam level the performance tests are mostly not >>>>> healthy anyhow. So >>>>> >>>>>>>>>>>>>> is there any high level blocker to switching them or is >>>>> it just someone >>>>> >>>>>>>>>>>>>> sitting down with each one? >>>>> >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>> >>>>> >>>>>>>>>>>>> I thought I could share my point of view here as I am >>>>> working >>>>> >>>>>>>>>>>>> on the Performance Test part for a while now. I wouldn't >>>>> say those are >>>>> >>>>>>>>>>>>> "mostly not healthy". The situation is as follows: >>>>> >>>>>>>>>>>>> >>>>> >>>>>>>>>>>>> - totally not working: Python, Spark performance tests >>>>> (don't >>>>> >>>>>>>>>>>>> know why, I'm not maintaining those tests. Should we >>>>> disable them?) >>>>> >>>>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance Test. >>>>> It's >>>>> >>>>>>>>>>>>> seems to be flaky mostly due to: >>>>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798 >>>>> >>>>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO, >>>>> Compressed >>>>> >>>>>>>>>>>>> Text, TFRecordIO >>>>> >>>>>>>>>>>>> >>>>> >>>>>>>>>>>>> Also, some test failures are caused due to pending PRs >>>>> (we >>>>> >>>>>>>>>>>>> sometimes run concrete tests to check if PRs won't break >>>>> them). This also >>>>> >>>>>>>>>>>>> causes failures sometimes. >>>>> >>>>>>>>>>>>> >>>>> >>>>>>>>>>>>> I can help with switching the Performance tests to >>>>> Gradle as >>>>> >>>>>>>>>>>>> this part seems to be free to take. >>>>> >>>>>>>>>>>>> >>>>> >>>>>>>>>>>>> >>>>> >>>>>>>>>>>>> Łukasz >>>>> >>>>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>> >>>>> >>>>>>> >>>>> >>>>> >>>>> >>> >>>>> > >>>>> >>>> >>>> >
