Ok so to be clear for any contributor (which is the goal of this thread): maven is still the main build system and no need to maintain gradle in PR then until beam switches.
Im more than fine with that. Le 22 mars 2018 18:22, "Alan Myrvold" <[email protected]> a écrit : > I think the investment in gradle is worthwhile, and incrementally we will > continue to make progress. From what I've seem, gradle is a good fit for > this project and a path to a faster, more reliable build system. > > pull/4812 <https://github.com/apache/beam/pull/4812> creates the release > artifacts, although it is not hooked up yet with authentication. > > I expect to help make incremental progress over the next month converting > some of the integration tests, but welcome incremental improvements from > others. > > > > On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau <[email protected]> > wrote: > >> >> >> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik <[email protected]>: >> >>> what do we do? "Gradle migration will happen incrementally." >>> >>> "last months prooved beam cant maintain 2 systems, easier with that >>> state is then to drop gradle since it is a 0 investment compared to the >>> opposite" >>> Its unfortunate that you feel this way but many people do not share your >>> opinion. >>> >> >> And a lot do so when a project is 50-50 it is time to act. >> >> Incrementally kind of means never (makes 4 months and nothing really >> changed in PRs and habits, gradle maintener(s) are still alone) >> >> >>> >>> >>> On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau < >>> [email protected]> wrote: >>> >>>> @Valentyn: concretely any user can PR and be part of that process so >>>> anyone can do it wrong (me first) >>>> @Luskasz, Hennking: fine but what do we do? last months prooved beam >>>> cant maintain 2 systems, easier with that state is then to drop gradle >>>> since it is a 0 investment compared to the opposite >>>> >>>> >>>> Romain Manni-Bucau >>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>> <http://rmannibucau.wordpress.com> | Github >>>> <https://github.com/rmannibucau> | LinkedIn >>>> <https://www.linkedin.com/in/rmannibucau> | Book >>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance> >>>> >>>> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik <[email protected]>: >>>> >>>>> Romain, from the previous discussions several people agreed that >>>>> running a fixit that migrated Maven to Gradle over a 1 or 2 day period was >>>>> worthwhile but there was nobody in the community with the time commitment >>>>> to organize and run it so the status quo plan remained where the Gradle >>>>> migration will happen incrementally. >>>>> >>>>> >>>>> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde <[email protected]> >>>>> wrote: >>>>> >>>>>> My understanding was the same as Ismaël's. I don't think breaking the >>>>>> build with a large known gaps (but not fully known cost) is practical. >>>>>> Also, most items in the jira are not even assigned yet. >>>>>> >>>>>> >>>>>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Not really Ismaël, this thread was about to do it at once and have 1 >>>>>>> day to fix it all. >>>>>>> >>>>>>> As mentionned at the very beginning nobody maintains the 2 system so >>>>>>> it must stop after months so either we drop maven or gradle *at once* >>>>>>> or we keep a state where each dev does what he wants and the build >>>>>>> system just doesn't work. >>>>>>> >>>>>>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía <[email protected]>: >>>>>>> >>>>>>>> I don't think that removing all maven descriptors was the expected >>>>>>>> path, no ? Or even a good idea at this moment. >>>>>>>> >>>>>>>> I understood that what we were going to do was to replace >>>>>>>> incrementally the CI until we cover the whole maven functionality >>>>>>>> and >>>>>>>> then remove it, from looking at the JIRA ticket >>>>>>>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far >>>>>>>> from >>>>>>>> covering the complete maven functionality in particular for the >>>>>>>> release part that could be the biggest pain point. >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau >>>>>>>> <[email protected]> wrote: >>>>>>>> > hey guys, >>>>>>>> > >>>>>>>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or >>>>>>>> on monday? >>>>>>>> > >>>>>>>> > >>>>>>>> > Romain Manni-Bucau >>>>>>>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>>>>>> > >>>>>>>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles <[email protected]>: >>>>>>>> >> >>>>>>>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik <[email protected]> >>>>>>>> wrote: >>>>>>>> >>> >>>>>>>> >>> Based upon your description it seems as though you would rather >>>>>>>> have a >>>>>>>> >>> way to run existing postcommits without it impacting >>>>>>>> dashboard/health >>>>>>>> >>> stats/notifications/.... (We have just run the PostCommits on >>>>>>>> PRs for >>>>>>>> >>> additional validation (like upgrading the Dataflow container >>>>>>>> image)). >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> Yes, that is exactly what I have described. >>>>>>>> >> >>>>>>>> >>> I don't think that keeping the current Java PreCommit as a >>>>>>>> proxy for the >>>>>>>> >>> the Java PostCommit is the right way to go but I also don't >>>>>>>> have the time to >>>>>>>> >>> implement what your actually asking for. >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> Mostly I thought this might be very easy based on the fact that >>>>>>>> they are >>>>>>>> >> nearly identical. If not, oh well. >>>>>>>> >> >>>>>>>> >> Kenn >>>>>>>> >> >>>>>>>> >> >>>>>>>> >>> It seems more likely that migrating the PostCommit to Gradle >>>>>>>> will be less >>>>>>>> >>> work then adding the functionality but your argument where the >>>>>>>> PreCommit is >>>>>>>> >>> a proxy for the Java PostCommit also applies to the >>>>>>>> ValidatesRunner >>>>>>>> >>> PostCommits and so forth requiring even more migration to >>>>>>>> happen before you >>>>>>>> >>> don't have to worry about maintaining Maven/breaking post >>>>>>>> commits. >>>>>>>> >>> >>>>>>>> >>> I'm fine with leaving both the Java/Gradle PreCommits running >>>>>>>> for now and >>>>>>>> >>> hopefully as more of the PostCommits are migrated off we will >>>>>>>> be able to >>>>>>>> >>> remove it. >>>>>>>> >>> >>>>>>>> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>> >>>>>>>> >>>> Separate history (for easy dashboarding, health stats, etc) and >>>>>>>> >>>> notification (email to dev@ for postcommits, nothing for >>>>>>>> precommits) for pre >>>>>>>> >>>> & post commit targets. >>>>>>>> >>>> >>>>>>>> >>>> A post commit failure is always a problem to be triaged at high >>>>>>>> >>>> priority, while a precommit failure is just a natural >>>>>>>> occurrence. >>>>>>>> >>>> >>>>>>>> >>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>> >>>>>>>> >>>>> Ken, I'm probably not seeing something but how does using the >>>>>>>> PreCommit >>>>>>>> >>>>> as a proxy improve upon just running the post commit via the >>>>>>>> phrase it >>>>>>>> >>>>> already supports ('Run Java PostCommit')? >>>>>>>> >>>>> >>>>>>>> >>>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles < >>>>>>>> [email protected]> >>>>>>>> >>>>> wrote: >>>>>>>> >>>>>> >>>>>>>> >>>>>> Indeed, we've already had the discussion a couple of times >>>>>>>> and I think >>>>>>>> >>>>>> the criteria are clearly met. Incremental progress is a good >>>>>>>> thing and we >>>>>>>> >>>>>> shouldn't block it. >>>>>>>> >>>>>> >>>>>>>> >>>>>> OTOH I see where Romain is coming from and I have a good >>>>>>>> example that >>>>>>>> >>>>>> supports a slightly different action. Consider >>>>>>>> >>>>>> https://github.com/apache/beam/pull/4740 which fixes some >>>>>>>> errors in how we >>>>>>>> >>>>>> use dependency mechanisms. >>>>>>>> >>>>>> >>>>>>>> >>>>>> This PR is green except that I need to fix some Maven pom >>>>>>>> slightly >>>>>>>> >>>>>> more. That is throwaway work. I would love to just not have >>>>>>>> to do it. But >>>>>>>> >>>>>> removing the precommit does not actually make the PR OK to >>>>>>>> merge. It would >>>>>>>> >>>>>> cause postcommits to fail. >>>>>>>> >>>>>> >>>>>>>> >>>>>> We can hope such situations are rare. I think I tend to be >>>>>>>> hit by this >>>>>>>> >>>>>> more often than most, as I work with the project build >>>>>>>> health quite a bit. >>>>>>>> >>>>>> >>>>>>>> >>>>>> Here is a proposal to support these things: instead of >>>>>>>> deleting the >>>>>>>> >>>>>> job in #4814, move it to not run automatically but only via >>>>>>>> a phrase. In >>>>>>>> >>>>>> fact, you could migrate it to be the manually-invoked >>>>>>>> version of the >>>>>>>> >>>>>> postcommit job as we've discussed a couple times. Then if >>>>>>>> someone is working >>>>>>>> >>>>>> on the build in something like #4740 they can invoke it >>>>>>>> manually. >>>>>>>> >>>>>> >>>>>>>> >>>>>> Kenn >>>>>>>> >>>>>> >>>>>>>> >>>>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> Based upon the criteria that was discussed on the mailing >>>>>>>> list[1], I >>>>>>>> >>>>>>> would agree with Kenn about merging PR/4814 (drop Java >>>>>>>> Maven precommit). >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> 1: >>>>>>>> >>>>>>> https://lists.apache.org/thread.html/ >>>>>>>> 7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@% >>>>>>>> 3Cdev.beam.apache.org%3E >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau >>>>>>>> >>>>>>> <[email protected]> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Hi Kenneth, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> For now maven covers the full needs of beam. If we start >>>>>>>> to have >>>>>>>> >>>>>>>> this kind of PR we become dependent of the 2 builds which >>>>>>>> is what this >>>>>>>> >>>>>>>> thread is about avoiding so tempted to say it must be a PR >>>>>>>> drop completely >>>>>>>> >>>>>>>> maven or nothing as mentionned before. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <[email protected]> >>>>>>>> a écrit : >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> I would like to briefly re-focus this discussion and >>>>>>>> suggest that >>>>>>>> >>>>>>>>> we merge https://github.com/apache/beam/pull/4814. >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> The only material objection I've heard is that it means >>>>>>>> the >>>>>>>> >>>>>>>>> precommit no longer tests exactly what is built for >>>>>>>> release. It is a valid >>>>>>>> >>>>>>>>> concern, but we have mvn postcommit so the coverage is >>>>>>>> not lost. The >>>>>>>> >>>>>>>>> benefits of quicker reviews and less Jenkins congestion >>>>>>>> seem worth it to me. >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> Kenn >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau >>>>>>>> >>>>>>>>> <[email protected]> wrote: >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> @Luskasz: not sure Im the best to host it since I know >>>>>>>> more gradle >>>>>>>> >>>>>>>>>> internals that user interface/ecosystem but happy to >>>>>>>> help. Will also require >>>>>>>> >>>>>>>>>> a "sudo" merger for this day to merge fixes asap - guess >>>>>>>> we can bypass >>>>>>>> >>>>>>>>>> reviews or have a fast cycle plan for this day to avoid >>>>>>>> it to be a week? >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <[email protected]> >>>>>>>> a écrit : >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way better >>>>>>>> than >>>>>>>> >>>>>>>>>> Maven, and what I mean is that I have added enough hints >>>>>>>> that it works OOTB >>>>>>>> >>>>>>>>>> already. The rest of my instructions are just how you >>>>>>>> should override >>>>>>>> >>>>>>>>>> IntelliJ's defaults to have a proper dev env - mostly >>>>>>>> just about storing >>>>>>>> >>>>>>>>>> files outside the git clone. >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> Hmm it is always super slow here and not as integrated >>>>>>>> as maven >>>>>>>> >>>>>>>>>> which is smooth thanks to the combination of idea build >>>>>>>> and maven plugin >>>>>>>> >>>>>>>>>> config read. Import is also faster cause it just reads >>>>>>>> meta and doesnt run >>>>>>>> >>>>>>>>>> anything. Hope it is a "a few times" issues at the >>>>>>>> moment but not yet sure. >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> @Łukasz - yea I just mean the ones that I find in my >>>>>>>> email and >>>>>>>> >>>>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/ >>>>>>>> which are the ones >>>>>>>> >>>>>>>>>> you classify as "totally not working". I would love to >>>>>>>> disable these. >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> On the subject of running things on a pending PR - we >>>>>>>> should not >>>>>>>> >>>>>>>>>> run postcommit jobs on PRs. We should make separate >>>>>>>> (optional) precommit >>>>>>>> >>>>>>>>>> jobs that run the same build commands. That will give a >>>>>>>> more clear history >>>>>>>> >>>>>>>>>> and allow trivial configuration of which jobs deserve >>>>>>>> alert emails and which >>>>>>>> >>>>>>>>>> are not a problem. This is easy but I've been waiting to >>>>>>>> do it after Gradle >>>>>>>> >>>>>>>>>> migration. >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik < >>>>>>>> [email protected]> >>>>>>>> >>>>>>>>>> wrote: >>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>>>>> Romain, would you like to host/plan/run the Gradle >>>>>>>> fixit day? >>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath >>>>>>>> >>>>>>>>>>> <[email protected]> wrote: >>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>> +1 for the general idea of fixit day/week for Gradle. >>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>> Agree with what Łukasz said. Some of these performance >>>>>>>> tests are >>>>>>>> >>>>>>>>>>>> new and are flaky due to other issues that were >>>>>>>> discovered during the >>>>>>>> >>>>>>>>>>>> process of adding the test. >>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>> I think the high level blocker is updating performance >>>>>>>> testing >>>>>>>> >>>>>>>>>>>> framework to use Gradle. This will involve adding >>>>>>>> Gradle-based logic for >>>>>>>> >>>>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and >>>>>>>> updating PerfKitBenchmarker >>>>>>>> >>>>>>>>>>>> to issue a Gradle command for running the benchmark >>>>>>>> [2]. First task will be >>>>>>>> >>>>>>>>>>>> to find the work needed here and updating the relevant >>>>>>>> JIRA [3]. >>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>> >>>>>>>>>>>> Cham >>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>> [1] >>>>>>>> >>>>>>>>>>>> https://github.com/apache/ >>>>>>>> beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79 >>>>>>>> >>>>>>>>>>>> [2] >>>>>>>> >>>>>>>>>>>> https://github.com/GoogleCloudPlatform/ >>>>>>>> PerfKitBenchmarker/blob/master/perfkitbenchmarker/ >>>>>>>> beam_benchmark_helper.py >>>>>>>> >>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251 >>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy >>>>>>>> >>>>>>>>>>>> <[email protected]> wrote: >>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles < >>>>>>>> [email protected]>: >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> Based on https://builds.apache.org/ >>>>>>>> view/A-D/view/Beam/ and our >>>>>>>> >>>>>>>>>>>>>> failure spam level the performance tests are mostly >>>>>>>> not healthy anyhow. So >>>>>>>> >>>>>>>>>>>>>> is there any high level blocker to switching them or >>>>>>>> is it just someone >>>>>>>> >>>>>>>>>>>>>> sitting down with each one? >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>> I thought I could share my point of view here as I am >>>>>>>> working >>>>>>>> >>>>>>>>>>>>> on the Performance Test part for a while now. I >>>>>>>> wouldn't say those are >>>>>>>> >>>>>>>>>>>>> "mostly not healthy". The situation is as follows: >>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>> - totally not working: Python, Spark performance >>>>>>>> tests (don't >>>>>>>> >>>>>>>>>>>>> know why, I'm not maintaining those tests. Should we >>>>>>>> disable them?) >>>>>>>> >>>>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance >>>>>>>> Test. It's >>>>>>>> >>>>>>>>>>>>> seems to be flaky mostly due to: >>>>>>>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798 >>>>>>>> >>>>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO, >>>>>>>> Compressed >>>>>>>> >>>>>>>>>>>>> Text, TFRecordIO >>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>> Also, some test failures are caused due to pending >>>>>>>> PRs (we >>>>>>>> >>>>>>>>>>>>> sometimes run concrete tests to check if PRs won't >>>>>>>> break them). This also >>>>>>>> >>>>>>>>>>>>> causes failures sometimes. >>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>> I can help with switching the Performance tests to >>>>>>>> Gradle as >>>>>>>> >>>>>>>>>>>>> this part seems to be free to take. >>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>> Łukasz >>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>> >>>>>>>> >>> >>>>>>>> > >>>>>>>> >>>>>>> >>>>>>> >>>> >>
