Thanks for organizing, Reuven. I too would like to see us move back to a single build system to reduce complexity. Count me in for the fixit.
On Thu, Mar 22, 2018 at 11:27 PM Reuven Lax <[email protected]> wrote: > I'll send an email tomorrow with a few proposed dates and set up a > burndown list of tasks. > > Reuven > > On Thu, Mar 22, 2018 at 11:24 PM Romain Manni-Bucau <[email protected]> > wrote: > >> Can work Reuven, where is the "todo" list? Thought we were done (= the >> replacement was not blocking the dev) multiple times due to other threads >> but reading this week mails it sounds like we are not yet here. >> >> >> Romain Manni-Bucau >> @rmannibucau <https://twitter.com/rmannibucau> | Blog >> <https://rmannibucau.metawerx.net/> | Old Blog >> <http://rmannibucau.wordpress.com> | Github >> <https://github.com/rmannibucau> | LinkedIn >> <https://www.linkedin.com/in/rmannibucau> | Book >> <https://www.packtpub.com/application-development/java-ee-8-high-performance> >> >> 2018-03-22 22:51 GMT+01:00 Reuven Lax <[email protected]>: >> >>> Let's back up for a second. >>> >>> Earlier in the thread we agreed to organize a community "fixit" day to >>> try and migrate remaining Maven items to Gradle. I had thought that Romain >>> had volunteered to run this, but reading back in the thread it appears that >>> I misunderstood this. I would suggest that we organize this first, and make >>> the concerted push to migrate the remaining items. After this is done, we >>> can evaluate the state we're left in and hold a process vote if necessary. >>> >>> I can volunteer to help coordinate this fixit. >>> >>> Reuven >>> >>> On Thu, Mar 22, 2018 at 1:35 PM Dan Halperin <[email protected]> >>> wrote: >>> >>>> On Thu, Mar 22, 2018 at 11:19 AM, Chamikara Jayalath < >>>> [email protected]> wrote: >>>> >>>>> I don't think incremental progress is a bad thing as long as we are >>>>> making progress towards the goal. Do we need better metrics (a weekly >>>>> email >>>>> ?) about the progress towards moving everything to Gradle ? I agree with >>>>> others who pointed out that there are many unresolved JIRAs and simply >>>>> deleting Maven artifacts could break many things (for example, performance >>>>> tests). >>>>> >>>> >>>> The problem does not seem to be incremental progress on its face, or a >>>> lack of metrics. >>>> >>>> The problem is that there are two build systems with separate features >>>> and issues, doubling (or worse) Jenkins cycles, mental effort, maintenance >>>> burden, etc. It hurts the community and casual contributors. >>>> >>>> As Luke suggested, >>>> > A process vote can be happen if the in-between state is too painful >>>> to maintain. >>>> >>>> Given that the in-between state has lasted so long, and there is it may >>>> be time. >>>> >>>> Dan >>>> >>>> >>>> >>>>> >>>>> Thanks, >>>>> Cham >>>>> >>>>> >>>>> On Thu, Mar 22, 2018 at 10:56 AM Romain Manni-Bucau < >>>>> [email protected]> wrote: >>>>> >>>>>> >>>>>> >>>>>> Le 22 mars 2018 18:49, "Dan Halperin" <[email protected]> a écrit : >>>>>> >>>>>> It seems that a few groups are talking past each other. >>>>>> >>>>>> * A sizable contingent is interested in a move to Gradle -- it shows >>>>>> promise, but the work is incomplete. >>>>>> * Another contingent noticing the large burden of maintaining >>>>>> multiple build systems. FWICT, both test suites have been broken quite a >>>>>> lot recently, mainly the gradle ones, which is a cost to the community. >>>>>> This is creating a barrier to entry for new contributors – especially >>>>>> those >>>>>> who don't work in the same room or do their primary development in a >>>>>> different repository. >>>>>> >>>>>> I don't see this situation being resolved to anyone's satisfaction >>>>>> until there's only one build system left. The onus is clearly on the >>>>>> Gradle >>>>>> promoters to finish the work. >>>>>> >>>>>> Luke made a suggestion 2.5 months ago that we should have a process >>>>>> vote if this situation is untenable. It seems like we're there. >>>>>> >>>>>> >>>>>> Yes but beam voted to move to gradle so we should but we shouldnt >>>>>> maintain 2 build systems for more than 2 months (weway overpassed that) >>>>>> and >>>>>> therefore the vote should be cancelled or validated by an action. >>>>>> >>>>>> I understand you want gradle but you dont want to pay the cost to >>>>>> move to gradle, it doesnt work for the project do please another option >>>>>> (rollbacking gradle or removing maven, all other options are negative for >>>>>> the project and a pain for committers and contributors whatever you >>>>>> think). >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> Dan >>>>>> >>>>>> On Thu, Mar 22, 2018 at 10:30 AM, Romain Manni-Bucau < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Ok so to be clear for any contributor (which is the goal of this >>>>>>> thread): maven is still the main build system and no need to maintain >>>>>>> gradle in PR then until beam switches. >>>>>>> >>>>>>> Im more than fine with that. >>>>>>> >>>>>>> Le 22 mars 2018 18:22, "Alan Myrvold" <[email protected]> a >>>>>>> écrit : >>>>>>> >>>>>>>> I think the investment in gradle is worthwhile, and incrementally >>>>>>>> we will continue to make progress. From what I've seem, gradle is a >>>>>>>> good >>>>>>>> fit for this project and a path to a faster, more reliable build >>>>>>>> system. >>>>>>>> >>>>>>>> pull/4812 <https://github.com/apache/beam/pull/4812> creates the >>>>>>>> release artifacts, although it is not hooked up yet with >>>>>>>> authentication. >>>>>>>> >>>>>>>> I expect to help make incremental progress over the next month >>>>>>>> converting some of the integration tests, but welcome incremental >>>>>>>> improvements from others. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik <[email protected]>: >>>>>>>>> >>>>>>>>>> what do we do? "Gradle migration will happen incrementally." >>>>>>>>>> >>>>>>>>>> "last months prooved beam cant maintain 2 systems, easier with >>>>>>>>>> that state is then to drop gradle since it is a 0 investment >>>>>>>>>> compared to >>>>>>>>>> the opposite" >>>>>>>>>> Its unfortunate that you feel this way but many people do not >>>>>>>>>> share your opinion. >>>>>>>>>> >>>>>>>>> >>>>>>>>> And a lot do so when a project is 50-50 it is time to act. >>>>>>>>> >>>>>>>>> Incrementally kind of means never (makes 4 months and nothing >>>>>>>>> really changed in PRs and habits, gradle maintener(s) are still alone) >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> @Valentyn: concretely any user can PR and be part of that >>>>>>>>>>> process so anyone can do it wrong (me first) >>>>>>>>>>> @Luskasz, Hennking: fine but what do we do? last months prooved >>>>>>>>>>> beam cant maintain 2 systems, easier with that state is then to >>>>>>>>>>> drop gradle >>>>>>>>>>> since it is a 0 investment compared to the opposite >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Romain Manni-Bucau >>>>>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>>>>>>> <http://rmannibucau.wordpress.com> | Github >>>>>>>>>>> <https://github.com/rmannibucau> | LinkedIn >>>>>>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book >>>>>>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance> >>>>>>>>>>> >>>>>>>>>>> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik <[email protected]>: >>>>>>>>>>> >>>>>>>>>>>> Romain, from the previous discussions several people agreed >>>>>>>>>>>> that running a fixit that migrated Maven to Gradle over a 1 or 2 >>>>>>>>>>>> day period >>>>>>>>>>>> was worthwhile but there was nobody in the community with the time >>>>>>>>>>>> commitment to organize and run it so the status quo plan remained >>>>>>>>>>>> where the >>>>>>>>>>>> Gradle migration will happen incrementally. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> My understanding was the same as Ismaël's. I don't think >>>>>>>>>>>>> breaking the build with a large known gaps (but not fully known >>>>>>>>>>>>> cost) is >>>>>>>>>>>>> practical. Also, most items in the jira are not even assigned yet. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Not really Ismaël, this thread was about to do it at once and >>>>>>>>>>>>>> have 1 day to fix it all. >>>>>>>>>>>>>> >>>>>>>>>>>>>> As mentionned at the very beginning nobody maintains the 2 >>>>>>>>>>>>>> system so it must stop after months so either we drop maven or >>>>>>>>>>>>>> gradle *at >>>>>>>>>>>>>> once* >>>>>>>>>>>>>> or we keep a state where each dev does what he wants and the >>>>>>>>>>>>>> build system just doesn't work. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía <[email protected]>: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I don't think that removing all maven descriptors was the >>>>>>>>>>>>>>> expected >>>>>>>>>>>>>>> path, no ? Or even a good idea at this moment. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I understood that what we were going to do was to replace >>>>>>>>>>>>>>> incrementally the CI until we cover the whole maven >>>>>>>>>>>>>>> functionality and >>>>>>>>>>>>>>> then remove it, from looking at the JIRA ticket >>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3249 we are >>>>>>>>>>>>>>> still far from >>>>>>>>>>>>>>> covering the complete maven functionality in particular for >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> release part that could be the biggest pain point. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau >>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>> > hey guys, >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > 2.4 is out, do we plan to drop all maven descriptors >>>>>>>>>>>>>>> tomorrow or on monday? >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Romain Manni-Bucau >>>>>>>>>>>>>>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles <[email protected] >>>>>>>>>>>>>>> >: >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> >>> Based upon your description it seems as though you would >>>>>>>>>>>>>>> rather have a >>>>>>>>>>>>>>> >>> way to run existing postcommits without it impacting >>>>>>>>>>>>>>> dashboard/health >>>>>>>>>>>>>>> >>> stats/notifications/.... (We have just run the >>>>>>>>>>>>>>> PostCommits on PRs for >>>>>>>>>>>>>>> >>> additional validation (like upgrading the Dataflow >>>>>>>>>>>>>>> container image)). >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> Yes, that is exactly what I have described. >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >>> I don't think that keeping the current Java PreCommit as >>>>>>>>>>>>>>> a proxy for the >>>>>>>>>>>>>>> >>> the Java PostCommit is the right way to go but I also >>>>>>>>>>>>>>> don't have the time to >>>>>>>>>>>>>>> >>> implement what your actually asking for. >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> Mostly I thought this might be very easy based on the >>>>>>>>>>>>>>> fact that they are >>>>>>>>>>>>>>> >> nearly identical. If not, oh well. >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> Kenn >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >>> It seems more likely that migrating the PostCommit to >>>>>>>>>>>>>>> Gradle will be less >>>>>>>>>>>>>>> >>> work then adding the functionality but your argument >>>>>>>>>>>>>>> where the PreCommit is >>>>>>>>>>>>>>> >>> a proxy for the Java PostCommit also applies to the >>>>>>>>>>>>>>> ValidatesRunner >>>>>>>>>>>>>>> >>> PostCommits and so forth requiring even more migration >>>>>>>>>>>>>>> to happen before you >>>>>>>>>>>>>>> >>> don't have to worry about maintaining Maven/breaking >>>>>>>>>>>>>>> post commits. >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> >>> I'm fine with leaving both the Java/Gradle PreCommits >>>>>>>>>>>>>>> running for now and >>>>>>>>>>>>>>> >>> hopefully as more of the PostCommits are migrated off we >>>>>>>>>>>>>>> will be able to >>>>>>>>>>>>>>> >>> remove it. >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> >>>> Separate history (for easy dashboarding, health stats, >>>>>>>>>>>>>>> etc) and >>>>>>>>>>>>>>> >>>> notification (email to dev@ for postcommits, nothing >>>>>>>>>>>>>>> for precommits) for pre >>>>>>>>>>>>>>> >>>> & post commit targets. >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> >>>> A post commit failure is always a problem to be triaged >>>>>>>>>>>>>>> at high >>>>>>>>>>>>>>> >>>> priority, while a precommit failure is just a natural >>>>>>>>>>>>>>> occurrence. >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> >>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>> >>>>> Ken, I'm probably not seeing something but how does >>>>>>>>>>>>>>> using the PreCommit >>>>>>>>>>>>>>> >>>>> as a proxy improve upon just running the post commit >>>>>>>>>>>>>>> via the phrase it >>>>>>>>>>>>>>> >>>>> already supports ('Run Java PostCommit')? >>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>> >>>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles < >>>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>> >>>>> wrote: >>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>> >>>>>> Indeed, we've already had the discussion a couple of >>>>>>>>>>>>>>> times and I think >>>>>>>>>>>>>>> >>>>>> the criteria are clearly met. Incremental progress is >>>>>>>>>>>>>>> a good thing and we >>>>>>>>>>>>>>> >>>>>> shouldn't block it. >>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>> >>>>>> OTOH I see where Romain is coming from and I have a >>>>>>>>>>>>>>> good example that >>>>>>>>>>>>>>> >>>>>> supports a slightly different action. Consider >>>>>>>>>>>>>>> >>>>>> https://github.com/apache/beam/pull/4740 which fixes >>>>>>>>>>>>>>> some errors in how we >>>>>>>>>>>>>>> >>>>>> use dependency mechanisms. >>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>> >>>>>> This PR is green except that I need to fix some Maven >>>>>>>>>>>>>>> pom slightly >>>>>>>>>>>>>>> >>>>>> more. That is throwaway work. I would love to just >>>>>>>>>>>>>>> not have to do it. But >>>>>>>>>>>>>>> >>>>>> removing the precommit does not actually make the PR >>>>>>>>>>>>>>> OK to merge. It would >>>>>>>>>>>>>>> >>>>>> cause postcommits to fail. >>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>> >>>>>> We can hope such situations are rare. I think I tend >>>>>>>>>>>>>>> to be hit by this >>>>>>>>>>>>>>> >>>>>> more often than most, as I work with the project >>>>>>>>>>>>>>> build health quite a bit. >>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>> >>>>>> Here is a proposal to support these things: instead >>>>>>>>>>>>>>> of deleting the >>>>>>>>>>>>>>> >>>>>> job in #4814, move it to not run automatically but >>>>>>>>>>>>>>> only via a phrase. In >>>>>>>>>>>>>>> >>>>>> fact, you could migrate it to be the manually-invoked >>>>>>>>>>>>>>> version of the >>>>>>>>>>>>>>> >>>>>> postcommit job as we've discussed a couple times. >>>>>>>>>>>>>>> Then if someone is working >>>>>>>>>>>>>>> >>>>>> on the build in something like #4740 they can invoke >>>>>>>>>>>>>>> it manually. >>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>> >>>>>> Kenn >>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>> >>>>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>> >>>>>>> Based upon the criteria that was discussed on the >>>>>>>>>>>>>>> mailing list[1], I >>>>>>>>>>>>>>> >>>>>>> would agree with Kenn about merging PR/4814 (drop >>>>>>>>>>>>>>> Java Maven precommit). >>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>> >>>>>>> 1: >>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>> https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E >>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>> >>>>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau >>>>>>>>>>>>>>> >>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>> >>>>>>>> Hi Kenneth, >>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>> >>>>>>>> For now maven covers the full needs of beam. If we >>>>>>>>>>>>>>> start to have >>>>>>>>>>>>>>> >>>>>>>> this kind of PR we become dependent of the 2 builds >>>>>>>>>>>>>>> which is what this >>>>>>>>>>>>>>> >>>>>>>> thread is about avoiding so tempted to say it must >>>>>>>>>>>>>>> be a PR drop completely >>>>>>>>>>>>>>> >>>>>>>> maven or nothing as mentionned before. >>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>> >>>>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" < >>>>>>>>>>>>>>> [email protected]> a écrit : >>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>> I would like to briefly re-focus this discussion >>>>>>>>>>>>>>> and suggest that >>>>>>>>>>>>>>> >>>>>>>>> we merge https://github.com/apache/beam/pull/4814. >>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>> The only material objection I've heard is that it >>>>>>>>>>>>>>> means the >>>>>>>>>>>>>>> >>>>>>>>> precommit no longer tests exactly what is built >>>>>>>>>>>>>>> for release. It is a valid >>>>>>>>>>>>>>> >>>>>>>>> concern, but we have mvn postcommit so the >>>>>>>>>>>>>>> coverage is not lost. The >>>>>>>>>>>>>>> >>>>>>>>> benefits of quicker reviews and less Jenkins >>>>>>>>>>>>>>> congestion seem worth it to me. >>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>> Kenn >>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau >>>>>>>>>>>>>>> >>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>> @Luskasz: not sure Im the best to host it since I >>>>>>>>>>>>>>> know more gradle >>>>>>>>>>>>>>> >>>>>>>>>> internals that user interface/ecosystem but happy >>>>>>>>>>>>>>> to help. Will also require >>>>>>>>>>>>>>> >>>>>>>>>> a "sudo" merger for this day to merge fixes asap >>>>>>>>>>>>>>> - guess we can bypass >>>>>>>>>>>>>>> >>>>>>>>>> reviews or have a fast cycle plan for this day to >>>>>>>>>>>>>>> avoid it to be a week? >>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" < >>>>>>>>>>>>>>> [email protected]> a écrit : >>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way >>>>>>>>>>>>>>> better than >>>>>>>>>>>>>>> >>>>>>>>>> Maven, and what I mean is that I have added >>>>>>>>>>>>>>> enough hints that it works OOTB >>>>>>>>>>>>>>> >>>>>>>>>> already. The rest of my instructions are just how >>>>>>>>>>>>>>> you should override >>>>>>>>>>>>>>> >>>>>>>>>> IntelliJ's defaults to have a proper dev env - >>>>>>>>>>>>>>> mostly just about storing >>>>>>>>>>>>>>> >>>>>>>>>> files outside the git clone. >>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>> Hmm it is always super slow here and not as >>>>>>>>>>>>>>> integrated as maven >>>>>>>>>>>>>>> >>>>>>>>>> which is smooth thanks to the combination of idea >>>>>>>>>>>>>>> build and maven plugin >>>>>>>>>>>>>>> >>>>>>>>>> config read. Import is also faster cause it just >>>>>>>>>>>>>>> reads meta and doesnt run >>>>>>>>>>>>>>> >>>>>>>>>> anything. Hope it is a "a few times" issues at >>>>>>>>>>>>>>> the moment but not yet sure. >>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>> @Łukasz - yea I just mean the ones that I find in >>>>>>>>>>>>>>> my email and >>>>>>>>>>>>>>> >>>>>>>>>> browsing >>>>>>>>>>>>>>> https://builds.apache.org/view/A-D/view/Beam/ which are the >>>>>>>>>>>>>>> ones >>>>>>>>>>>>>>> >>>>>>>>>> you classify as "totally not working". I would >>>>>>>>>>>>>>> love to disable these. >>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>> On the subject of running things on a pending PR >>>>>>>>>>>>>>> - we should not >>>>>>>>>>>>>>> >>>>>>>>>> run postcommit jobs on PRs. We should make >>>>>>>>>>>>>>> separate (optional) precommit >>>>>>>>>>>>>>> >>>>>>>>>> jobs that run the same build commands. That will >>>>>>>>>>>>>>> give a more clear history >>>>>>>>>>>>>>> >>>>>>>>>> and allow trivial configuration of which jobs >>>>>>>>>>>>>>> deserve alert emails and which >>>>>>>>>>>>>>> >>>>>>>>>> are not a problem. This is easy but I've been >>>>>>>>>>>>>>> waiting to do it after Gradle >>>>>>>>>>>>>>> >>>>>>>>>> migration. >>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik < >>>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>> Romain, would you like to host/plan/run the >>>>>>>>>>>>>>> Gradle fixit day? >>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara >>>>>>>>>>>>>>> Jayalath >>>>>>>>>>>>>>> >>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>> +1 for the general idea of fixit day/week for >>>>>>>>>>>>>>> Gradle. >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>> Agree with what Łukasz said. Some of these >>>>>>>>>>>>>>> performance tests are >>>>>>>>>>>>>>> >>>>>>>>>>>> new and are flaky due to other issues that were >>>>>>>>>>>>>>> discovered during the >>>>>>>>>>>>>>> >>>>>>>>>>>> process of adding the test. >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>> I think the high level blocker is updating >>>>>>>>>>>>>>> performance testing >>>>>>>>>>>>>>> >>>>>>>>>>>> framework to use Gradle. This will involve >>>>>>>>>>>>>>> adding Gradle-based logic for >>>>>>>>>>>>>>> >>>>>>>>>>>> invoking PerfKitBenchmaker, for example [1], >>>>>>>>>>>>>>> and updating PerfKitBenchmarker >>>>>>>>>>>>>>> >>>>>>>>>>>> to issue a Gradle command for running the >>>>>>>>>>>>>>> benchmark [2]. First task will be >>>>>>>>>>>>>>> >>>>>>>>>>>> to find the work needed here and updating the >>>>>>>>>>>>>>> relevant JIRA [3]. >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> >>>>>>>>>>>> Cham >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>> [1] >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79 >>>>>>>>>>>>>>> >>>>>>>>>>>> [2] >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py >>>>>>>>>>>>>>> >>>>>>>>>>>> [3] >>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3251 >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy >>>>>>>>>>>>>>> >>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles < >>>>>>>>>>>>>>> [email protected]>: >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> Based on >>>>>>>>>>>>>>> https://builds.apache.org/view/A-D/view/Beam/ and our >>>>>>>>>>>>>>> >>>>>>>>>>>>>> failure spam level the performance tests are >>>>>>>>>>>>>>> mostly not healthy anyhow. So >>>>>>>>>>>>>>> >>>>>>>>>>>>>> is there any high level blocker to switching >>>>>>>>>>>>>>> them or is it just someone >>>>>>>>>>>>>>> >>>>>>>>>>>>>> sitting down with each one? >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> I thought I could share my point of view here >>>>>>>>>>>>>>> as I am working >>>>>>>>>>>>>>> >>>>>>>>>>>>> on the Performance Test part for a while now. >>>>>>>>>>>>>>> I wouldn't say those are >>>>>>>>>>>>>>> >>>>>>>>>>>>> "mostly not healthy". The situation is as >>>>>>>>>>>>>>> follows: >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> - totally not working: Python, Spark >>>>>>>>>>>>>>> performance tests (don't >>>>>>>>>>>>>>> >>>>>>>>>>>>> know why, I'm not maintaining those tests. >>>>>>>>>>>>>>> Should we disable them?) >>>>>>>>>>>>>>> >>>>>>>>>>>>> - flaky: the recently re-enabled JDBC >>>>>>>>>>>>>>> Performance Test. It's >>>>>>>>>>>>>>> >>>>>>>>>>>>> seems to be flaky mostly due to: >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798 >>>>>>>>>>>>>>> >>>>>>>>>>>>> - working well, rarely failing: AvroIO, >>>>>>>>>>>>>>> TextIO, Compressed >>>>>>>>>>>>>>> >>>>>>>>>>>>> Text, TFRecordIO >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> Also, some test failures are caused due to >>>>>>>>>>>>>>> pending PRs (we >>>>>>>>>>>>>>> >>>>>>>>>>>>> sometimes run concrete tests to check if PRs >>>>>>>>>>>>>>> won't break them). This also >>>>>>>>>>>>>>> >>>>>>>>>>>>> causes failures sometimes. >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> I can help with switching the Performance >>>>>>>>>>>>>>> tests to Gradle as >>>>>>>>>>>>>>> >>>>>>>>>>>>> this part seems to be free to take. >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> Łukasz >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>> >>>>>> >>>> >>>> >> -- Got feedback? http://go/swegner-feedback
