My understanding was the same as Ismaël's. I don't think breaking the build with a large known gaps (but not fully known cost) is practical. Also, most items in the jira are not even assigned yet.
On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > Not really Ismaël, this thread was about to do it at once and have 1 day > to fix it all. > > As mentionned at the very beginning nobody maintains the 2 system so it > must stop after months so either we drop maven or gradle *at once* > or we keep a state where each dev does what he wants and the build system > just doesn't work. > > 2018-03-22 15:42 GMT+01:00 Ismaël Mejía <ieme...@gmail.com>: > >> I don't think that removing all maven descriptors was the expected >> path, no ? Or even a good idea at this moment. >> >> I understood that what we were going to do was to replace >> incrementally the CI until we cover the whole maven functionality and >> then remove it, from looking at the JIRA ticket >> https://issues.apache.org/jira/browse/BEAM-3249 we are still far from >> covering the complete maven functionality in particular for the >> release part that could be the biggest pain point. >> >> >> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau >> <rmannibu...@gmail.com> wrote: >> > hey guys, >> > >> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on >> monday? >> > >> > >> > Romain Manni-Bucau >> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >> > >> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles <k...@google.com>: >> >> >> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik <lc...@google.com> wrote: >> >>> >> >>> Based upon your description it seems as though you would rather have a >> >>> way to run existing postcommits without it impacting dashboard/health >> >>> stats/notifications/.... (We have just run the PostCommits on PRs for >> >>> additional validation (like upgrading the Dataflow container image)). >> >> >> >> >> >> Yes, that is exactly what I have described. >> >> >> >>> I don't think that keeping the current Java PreCommit as a proxy for >> the >> >>> the Java PostCommit is the right way to go but I also don't have the >> time to >> >>> implement what your actually asking for. >> >> >> >> >> >> Mostly I thought this might be very easy based on the fact that they >> are >> >> nearly identical. If not, oh well. >> >> >> >> Kenn >> >> >> >> >> >>> It seems more likely that migrating the PostCommit to Gradle will be >> less >> >>> work then adding the functionality but your argument where the >> PreCommit is >> >>> a proxy for the Java PostCommit also applies to the ValidatesRunner >> >>> PostCommits and so forth requiring even more migration to happen >> before you >> >>> don't have to worry about maintaining Maven/breaking post commits. >> >>> >> >>> I'm fine with leaving both the Java/Gradle PreCommits running for now >> and >> >>> hopefully as more of the PostCommits are migrated off we will be able >> to >> >>> remove it. >> >>> >> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles <k...@google.com> >> wrote: >> >>>> >> >>>> Separate history (for easy dashboarding, health stats, etc) and >> >>>> notification (email to dev@ for postcommits, nothing for >> precommits) for pre >> >>>> & post commit targets. >> >>>> >> >>>> A post commit failure is always a problem to be triaged at high >> >>>> priority, while a precommit failure is just a natural occurrence. >> >>>> >> >>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <lc...@google.com> >> wrote: >> >>>>> >> >>>>> Ken, I'm probably not seeing something but how does using the >> PreCommit >> >>>>> as a proxy improve upon just running the post commit via the phrase >> it >> >>>>> already supports ('Run Java PostCommit')? >> >>>>> >> >>>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles <k...@google.com> >> >>>>> wrote: >> >>>>>> >> >>>>>> Indeed, we've already had the discussion a couple of times and I >> think >> >>>>>> the criteria are clearly met. Incremental progress is a good thing >> and we >> >>>>>> shouldn't block it. >> >>>>>> >> >>>>>> OTOH I see where Romain is coming from and I have a good example >> that >> >>>>>> supports a slightly different action. Consider >> >>>>>> https://github.com/apache/beam/pull/4740 which fixes some errors >> in how we >> >>>>>> use dependency mechanisms. >> >>>>>> >> >>>>>> This PR is green except that I need to fix some Maven pom slightly >> >>>>>> more. That is throwaway work. I would love to just not have to do >> it. But >> >>>>>> removing the precommit does not actually make the PR OK to merge. >> It would >> >>>>>> cause postcommits to fail. >> >>>>>> >> >>>>>> We can hope such situations are rare. I think I tend to be hit by >> this >> >>>>>> more often than most, as I work with the project build health >> quite a bit. >> >>>>>> >> >>>>>> Here is a proposal to support these things: instead of deleting the >> >>>>>> job in #4814, move it to not run automatically but only via a >> phrase. In >> >>>>>> fact, you could migrate it to be the manually-invoked version of >> the >> >>>>>> postcommit job as we've discussed a couple times. Then if someone >> is working >> >>>>>> on the build in something like #4740 they can invoke it manually. >> >>>>>> >> >>>>>> Kenn >> >>>>>> >> >>>>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <lc...@google.com> >> wrote: >> >>>>>>> >> >>>>>>> Based upon the criteria that was discussed on the mailing >> list[1], I >> >>>>>>> would agree with Kenn about merging PR/4814 (drop Java Maven >> precommit). >> >>>>>>> >> >>>>>>> 1: >> >>>>>>> >> https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E >> >>>>>>> >> >>>>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau >> >>>>>>> <rmannibu...@gmail.com> wrote: >> >>>>>>>> >> >>>>>>>> Hi Kenneth, >> >>>>>>>> >> >>>>>>>> For now maven covers the full needs of beam. If we start to have >> >>>>>>>> this kind of PR we become dependent of the 2 builds which is >> what this >> >>>>>>>> thread is about avoiding so tempted to say it must be a PR drop >> completely >> >>>>>>>> maven or nothing as mentionned before. >> >>>>>>>> >> >>>>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <k...@google.com> a >> écrit : >> >>>>>>>>> >> >>>>>>>>> I would like to briefly re-focus this discussion and suggest >> that >> >>>>>>>>> we merge https://github.com/apache/beam/pull/4814. >> >>>>>>>>> >> >>>>>>>>> The only material objection I've heard is that it means the >> >>>>>>>>> precommit no longer tests exactly what is built for release. It >> is a valid >> >>>>>>>>> concern, but we have mvn postcommit so the coverage is not >> lost. The >> >>>>>>>>> benefits of quicker reviews and less Jenkins congestion seem >> worth it to me. >> >>>>>>>>> >> >>>>>>>>> Kenn >> >>>>>>>>> >> >>>>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau >> >>>>>>>>> <rmannibu...@gmail.com> wrote: >> >>>>>>>>>> >> >>>>>>>>>> @Luskasz: not sure Im the best to host it since I know more >> gradle >> >>>>>>>>>> internals that user interface/ecosystem but happy to help. >> Will also require >> >>>>>>>>>> a "sudo" merger for this day to merge fixes asap - guess we >> can bypass >> >>>>>>>>>> reviews or have a fast cycle plan for this day to avoid it to >> be a week? >> >>>>>>>>>> >> >>>>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <k...@google.com> a >> écrit : >> >>>>>>>>>> >> >>>>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way better than >> >>>>>>>>>> Maven, and what I mean is that I have added enough hints that >> it works OOTB >> >>>>>>>>>> already. The rest of my instructions are just how you should >> override >> >>>>>>>>>> IntelliJ's defaults to have a proper dev env - mostly just >> about storing >> >>>>>>>>>> files outside the git clone. >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> Hmm it is always super slow here and not as integrated as maven >> >>>>>>>>>> which is smooth thanks to the combination of idea build and >> maven plugin >> >>>>>>>>>> config read. Import is also faster cause it just reads meta >> and doesnt run >> >>>>>>>>>> anything. Hope it is a "a few times" issues at the moment but >> not yet sure. >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> @Łukasz - yea I just mean the ones that I find in my email and >> >>>>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/ which >> are the ones >> >>>>>>>>>> you classify as "totally not working". I would love to disable >> these. >> >>>>>>>>>> >> >>>>>>>>>> On the subject of running things on a pending PR - we should >> not >> >>>>>>>>>> run postcommit jobs on PRs. We should make separate (optional) >> precommit >> >>>>>>>>>> jobs that run the same build commands. That will give a more >> clear history >> >>>>>>>>>> and allow trivial configuration of which jobs deserve alert >> emails and which >> >>>>>>>>>> are not a problem. This is easy but I've been waiting to do it >> after Gradle >> >>>>>>>>>> migration. >> >>>>>>>>>> >> >>>>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <lc...@google.com> >> >>>>>>>>>> wrote: >> >>>>>>>>>>> >> >>>>>>>>>>> Romain, would you like to host/plan/run the Gradle fixit day? >> >>>>>>>>>>> >> >>>>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath >> >>>>>>>>>>> <chamik...@google.com> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>> +1 for the general idea of fixit day/week for Gradle. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Agree with what Łukasz said. Some of these performance tests >> are >> >>>>>>>>>>>> new and are flaky due to other issues that were discovered >> during the >> >>>>>>>>>>>> process of adding the test. >> >>>>>>>>>>>> >> >>>>>>>>>>>> I think the high level blocker is updating performance >> testing >> >>>>>>>>>>>> framework to use Gradle. This will involve adding >> Gradle-based logic for >> >>>>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and updating >> PerfKitBenchmarker >> >>>>>>>>>>>> to issue a Gradle command for running the benchmark [2]. >> First task will be >> >>>>>>>>>>>> to find the work needed here and updating the relevant JIRA >> [3]. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Thanks, >> >>>>>>>>>>>> Cham >> >>>>>>>>>>>> >> >>>>>>>>>>>> [1] >> >>>>>>>>>>>> >> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79 >> >>>>>>>>>>>> [2] >> >>>>>>>>>>>> >> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py >> >>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251 >> >>>>>>>>>>>> >> >>>>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy >> >>>>>>>>>>>> <lukasz.gaj...@gmail.com> wrote: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <k...@google.com >> >: >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/ >> and our >> >>>>>>>>>>>>>> failure spam level the performance tests are mostly not >> healthy anyhow. So >> >>>>>>>>>>>>>> is there any high level blocker to switching them or is it >> just someone >> >>>>>>>>>>>>>> sitting down with each one? >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> I thought I could share my point of view here as I am >> working >> >>>>>>>>>>>>> on the Performance Test part for a while now. I wouldn't >> say those are >> >>>>>>>>>>>>> "mostly not healthy". The situation is as follows: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> - totally not working: Python, Spark performance tests >> (don't >> >>>>>>>>>>>>> know why, I'm not maintaining those tests. Should we >> disable them?) >> >>>>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance Test. It's >> >>>>>>>>>>>>> seems to be flaky mostly due to: >> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798 >> >>>>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO, Compressed >> >>>>>>>>>>>>> Text, TFRecordIO >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Also, some test failures are caused due to pending PRs (we >> >>>>>>>>>>>>> sometimes run concrete tests to check if PRs won't break >> them). This also >> >>>>>>>>>>>>> causes failures sometimes. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> I can help with switching the Performance tests to Gradle as >> >>>>>>>>>>>>> this part seems to be free to take. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Łukasz >> >>>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>> >> >>>>> >> >>> >> > >> > >
