Not really Ismaël, this thread was about to do it at once and have 1 day to fix it all.
As mentionned at the very beginning nobody maintains the 2 system so it must stop after months so either we drop maven or gradle *at once* or we keep a state where each dev does what he wants and the build system just doesn't work. 2018-03-22 15:42 GMT+01:00 Ismaël Mejía <[email protected]>: > I don't think that removing all maven descriptors was the expected > path, no ? Or even a good idea at this moment. > > I understood that what we were going to do was to replace > incrementally the CI until we cover the whole maven functionality and > then remove it, from looking at the JIRA ticket > https://issues.apache.org/jira/browse/BEAM-3249 we are still far from > covering the complete maven functionality in particular for the > release part that could be the biggest pain point. > > > On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau > <[email protected]> wrote: > > hey guys, > > > > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on > monday? > > > > > > Romain Manni-Bucau > > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book > > > > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles <[email protected]>: > >> > >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik <[email protected]> wrote: > >>> > >>> Based upon your description it seems as though you would rather have a > >>> way to run existing postcommits without it impacting dashboard/health > >>> stats/notifications/.... (We have just run the PostCommits on PRs for > >>> additional validation (like upgrading the Dataflow container image)). > >> > >> > >> Yes, that is exactly what I have described. > >> > >>> I don't think that keeping the current Java PreCommit as a proxy for > the > >>> the Java PostCommit is the right way to go but I also don't have the > time to > >>> implement what your actually asking for. > >> > >> > >> Mostly I thought this might be very easy based on the fact that they are > >> nearly identical. If not, oh well. > >> > >> Kenn > >> > >> > >>> It seems more likely that migrating the PostCommit to Gradle will be > less > >>> work then adding the functionality but your argument where the > PreCommit is > >>> a proxy for the Java PostCommit also applies to the ValidatesRunner > >>> PostCommits and so forth requiring even more migration to happen > before you > >>> don't have to worry about maintaining Maven/breaking post commits. > >>> > >>> I'm fine with leaving both the Java/Gradle PreCommits running for now > and > >>> hopefully as more of the PostCommits are migrated off we will be able > to > >>> remove it. > >>> > >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles <[email protected]> > wrote: > >>>> > >>>> Separate history (for easy dashboarding, health stats, etc) and > >>>> notification (email to dev@ for postcommits, nothing for precommits) > for pre > >>>> & post commit targets. > >>>> > >>>> A post commit failure is always a problem to be triaged at high > >>>> priority, while a precommit failure is just a natural occurrence. > >>>> > >>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <[email protected]> wrote: > >>>>> > >>>>> Ken, I'm probably not seeing something but how does using the > PreCommit > >>>>> as a proxy improve upon just running the post commit via the phrase > it > >>>>> already supports ('Run Java PostCommit')? > >>>>> > >>>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles <[email protected]> > >>>>> wrote: > >>>>>> > >>>>>> Indeed, we've already had the discussion a couple of times and I > think > >>>>>> the criteria are clearly met. Incremental progress is a good thing > and we > >>>>>> shouldn't block it. > >>>>>> > >>>>>> OTOH I see where Romain is coming from and I have a good example > that > >>>>>> supports a slightly different action. Consider > >>>>>> https://github.com/apache/beam/pull/4740 which fixes some errors > in how we > >>>>>> use dependency mechanisms. > >>>>>> > >>>>>> This PR is green except that I need to fix some Maven pom slightly > >>>>>> more. That is throwaway work. I would love to just not have to do > it. But > >>>>>> removing the precommit does not actually make the PR OK to merge. > It would > >>>>>> cause postcommits to fail. > >>>>>> > >>>>>> We can hope such situations are rare. I think I tend to be hit by > this > >>>>>> more often than most, as I work with the project build health quite > a bit. > >>>>>> > >>>>>> Here is a proposal to support these things: instead of deleting the > >>>>>> job in #4814, move it to not run automatically but only via a > phrase. In > >>>>>> fact, you could migrate it to be the manually-invoked version of the > >>>>>> postcommit job as we've discussed a couple times. Then if someone > is working > >>>>>> on the build in something like #4740 they can invoke it manually. > >>>>>> > >>>>>> Kenn > >>>>>> > >>>>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <[email protected]> > wrote: > >>>>>>> > >>>>>>> Based upon the criteria that was discussed on the mailing list[1], > I > >>>>>>> would agree with Kenn about merging PR/4814 (drop Java Maven > precommit). > >>>>>>> > >>>>>>> 1: > >>>>>>> https://lists.apache.org/thread.html/ > 7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@% > 3Cdev.beam.apache.org%3E > >>>>>>> > >>>>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau > >>>>>>> <[email protected]> wrote: > >>>>>>>> > >>>>>>>> Hi Kenneth, > >>>>>>>> > >>>>>>>> For now maven covers the full needs of beam. If we start to have > >>>>>>>> this kind of PR we become dependent of the 2 builds which is what > this > >>>>>>>> thread is about avoiding so tempted to say it must be a PR drop > completely > >>>>>>>> maven or nothing as mentionned before. > >>>>>>>> > >>>>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <[email protected]> a écrit > : > >>>>>>>>> > >>>>>>>>> I would like to briefly re-focus this discussion and suggest that > >>>>>>>>> we merge https://github.com/apache/beam/pull/4814. > >>>>>>>>> > >>>>>>>>> The only material objection I've heard is that it means the > >>>>>>>>> precommit no longer tests exactly what is built for release. It > is a valid > >>>>>>>>> concern, but we have mvn postcommit so the coverage is not lost. > The > >>>>>>>>> benefits of quicker reviews and less Jenkins congestion seem > worth it to me. > >>>>>>>>> > >>>>>>>>> Kenn > >>>>>>>>> > >>>>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau > >>>>>>>>> <[email protected]> wrote: > >>>>>>>>>> > >>>>>>>>>> @Luskasz: not sure Im the best to host it since I know more > gradle > >>>>>>>>>> internals that user interface/ecosystem but happy to help. Will > also require > >>>>>>>>>> a "sudo" merger for this day to merge fixes asap - guess we can > bypass > >>>>>>>>>> reviews or have a fast cycle plan for this day to avoid it to > be a week? > >>>>>>>>>> > >>>>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <[email protected]> a > écrit : > >>>>>>>>>> > >>>>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way better than > >>>>>>>>>> Maven, and what I mean is that I have added enough hints that > it works OOTB > >>>>>>>>>> already. The rest of my instructions are just how you should > override > >>>>>>>>>> IntelliJ's defaults to have a proper dev env - mostly just > about storing > >>>>>>>>>> files outside the git clone. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Hmm it is always super slow here and not as integrated as maven > >>>>>>>>>> which is smooth thanks to the combination of idea build and > maven plugin > >>>>>>>>>> config read. Import is also faster cause it just reads meta and > doesnt run > >>>>>>>>>> anything. Hope it is a "a few times" issues at the moment but > not yet sure. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> @Łukasz - yea I just mean the ones that I find in my email and > >>>>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/ which > are the ones > >>>>>>>>>> you classify as "totally not working". I would love to disable > these. > >>>>>>>>>> > >>>>>>>>>> On the subject of running things on a pending PR - we should not > >>>>>>>>>> run postcommit jobs on PRs. We should make separate (optional) > precommit > >>>>>>>>>> jobs that run the same build commands. That will give a more > clear history > >>>>>>>>>> and allow trivial configuration of which jobs deserve alert > emails and which > >>>>>>>>>> are not a problem. This is easy but I've been waiting to do it > after Gradle > >>>>>>>>>> migration. > >>>>>>>>>> > >>>>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <[email protected]> > >>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Romain, would you like to host/plan/run the Gradle fixit day? > >>>>>>>>>>> > >>>>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath > >>>>>>>>>>> <[email protected]> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> +1 for the general idea of fixit day/week for Gradle. > >>>>>>>>>>>> > >>>>>>>>>>>> Agree with what Łukasz said. Some of these performance tests > are > >>>>>>>>>>>> new and are flaky due to other issues that were discovered > during the > >>>>>>>>>>>> process of adding the test. > >>>>>>>>>>>> > >>>>>>>>>>>> I think the high level blocker is updating performance testing > >>>>>>>>>>>> framework to use Gradle. This will involve adding > Gradle-based logic for > >>>>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and updating > PerfKitBenchmarker > >>>>>>>>>>>> to issue a Gradle command for running the benchmark [2]. > First task will be > >>>>>>>>>>>> to find the work needed here and updating the relevant JIRA > [3]. > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> Cham > >>>>>>>>>>>> > >>>>>>>>>>>> [1] > >>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/ > google-cloud-platform/pom.xml#L79 > >>>>>>>>>>>> [2] > >>>>>>>>>>>> https://github.com/GoogleCloudPlatform/ > PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py > >>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251 > >>>>>>>>>>>> > >>>>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy > >>>>>>>>>>>> <[email protected]> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <[email protected]>: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and > our > >>>>>>>>>>>>>> failure spam level the performance tests are mostly not > healthy anyhow. So > >>>>>>>>>>>>>> is there any high level blocker to switching them or is it > just someone > >>>>>>>>>>>>>> sitting down with each one? > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> I thought I could share my point of view here as I am working > >>>>>>>>>>>>> on the Performance Test part for a while now. I wouldn't say > those are > >>>>>>>>>>>>> "mostly not healthy". The situation is as follows: > >>>>>>>>>>>>> > >>>>>>>>>>>>> - totally not working: Python, Spark performance tests (don't > >>>>>>>>>>>>> know why, I'm not maintaining those tests. Should we disable > them?) > >>>>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance Test. It's > >>>>>>>>>>>>> seems to be flaky mostly due to: > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798 > >>>>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO, Compressed > >>>>>>>>>>>>> Text, TFRecordIO > >>>>>>>>>>>>> > >>>>>>>>>>>>> Also, some test failures are caused due to pending PRs (we > >>>>>>>>>>>>> sometimes run concrete tests to check if PRs won't break > them). This also > >>>>>>>>>>>>> causes failures sometimes. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I can help with switching the Performance tests to Gradle as > >>>>>>>>>>>>> this part seems to be free to take. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Łukasz > >>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>> > >>> > > >
