> Also on our contributor site [3] we recommend ensuring to be able to run all tests with 'gradlew check' which is not to far away from full build. Probably no one would expect a full build to fail here, which again makes me think we need an equivalent Job on Jenkins here?
I think this is a really important point. When I first started, I wasted a decent amount of time trying to make `./gradlew check` work because it's mentioned in the contributor guide. Eventually my teammates told me not to worry about it, but it would have been nice to get that warm fuzzy of completing a full build (And also, had I not been part of a team with Beam experience it could have been very discouraging). Maybe we should put a note there pointing out that not all checks are expected to pass on master? We could suggest new contributors run the checks from the latest release tag instead. Brian On Thu, Mar 14, 2019 at 4:53 AM Michael Luckey <[email protected]> wrote: > Hi, > > +1 for getting a controlled environment. I am a bit scared to rely on > developers configuration to actually build release artefacts. For instance > during my release process testing I used a docker image, only to realise > after quite a few cycles, that default locale was off which leads to java > using ascii file encodings. This is most likely something we do not want to > happen in real live. And the plain fact, that probably every devel uses a > different jdk build makes - apart from other differences -, the release > build not reproducible, no? > > On the other hand, I finally managed to find that Jenkins build, which > actually executes a full build [1]. Unfortunately this also seems to be > consistently failing since March, 1. [2] > > [1] https://builds.apache.org/job/beam_Release_NightlySnapshot/ > [2] > https://builds.apache.org/job/beam_Release_NightlySnapshot/buildTimeTrend > > On Mon, Mar 11, 2019 at 9:41 PM Ahmet Altay <[email protected]> wrote: > >> >> >> On Mon, Mar 11, 2019 at 7:03 AM Michael Luckey <[email protected]> >> wrote: >> >>> >>> >>> On Mon, Mar 11, 2019 at 3:51 AM Kenneth Knowles <[email protected]> wrote: >>> >>>> I have never actually tried to run a full build recently. It takes a >>>> long time and usually isn't what is needed for a particular change. FWIW I >>>> view Beam at this point as a mini-monorepo, so each directory and target >>>> can be healthy/unhealthy on its own. >>>> >>> >>> Fair Point. Totally agree. >>> >>> >>>> >>>> But it does seem like we should at least know what is unhealthy and >>>> why. Have you been filing Jiras about the failures? Are they predictable? >>>> Are they targets that pass in Jenkins but not in vanilla build? That would >>>> mean our Jenkins environment is too rich and should be slimmed down >>>> probably. >>>> >>>> Kenn >>>> >>> >>> Unfortunately those failures are not really predictable. Usually, I >>> start with plain './gradlew build' and keep adding some '-x >>> :beam-sdks-python:testPy2Gcp -x :beam-sdks-python:testPython' until build >>> succeeds. Afterwards it seems to be possible to remove this exclusions step >>> by step, thereby filling the build cache, which on next reruns might have >>> some impact on how tasks are executed. >>> >>> Most of failures are python related. Had not much success getting into >>> those. From time to time I see 'seemingly' similar failures on Jenkins, but >>> tracing on python is more difficult coming from the more java background. >>> Also using Mac I believe to remember that preinstalled python had some >>> issues/differences compared with private installs. Others are those >>> Elasticsearch - which were worked on lately - and ClickHouse tests which >>> seem to be still flaky. >>> >> >>> So I mainly blamed my setup and did not yet have the time to further >>> track those failures down. But >>> >>> As I did use a vanilla system and was not able to get beam to build, i >>> got thinking about >>> >>> 1. The release process >>> The release manager has lot of stuff to take care for, but is also >>> supposed to run a full gradle build on her local machine [1]. Apart from >>> that being a long lasting task, if it keeps failing this puts additional >>> burden on the release manager. So I was wondering, why we can not push that >>> to Jenkins as we do with all these other tests [2]. Here I did not find any >>> existing Job doing such, so wanted to ask for feedback here. >>> >>> If a full build has to be run - and of course it makes some sense on >>> release - I would suggest to get that running on a regular base on Jenkins >>> just to ensure not to be surprised during release. And as a sideeffect >>> enable the release manager to also delegate this to Jenkins to free her >>> time (and dev box). >>> >> >> +1, this will be great. Quite often we end up catching issues right when >> we are doing the release. I would one up this request and suggest a Jenkins >> job running most of the release process as much as possible to avoid last >> minute surprises. >> >> Also, I wonder if we could build our releases in a controlled environment >> (e.g. container), that way the release would be more reproducible and the >> release manager would have to spend less time setting their environment >> (e.g. environment for building python in your case). >> >> >>> >>> Until now I had not yet have the time to investigate, whether all this >>> pre/Post/elseJobs cover all our tests on all modules. Hoped someone else >>> could point to the list of jobs which cover all tests. >>> >>> 2. Those newcomers >>> As a naive newcomer to a project I usually deal with those jars. After >>> problems arose, I might download corresponding sources and try to >>> investigate. First thing I usually do here is a make/build/install to get >>> those private builds in. Also on our contributor site [3] we recommend >>> ensuring to be able to run all tests with 'gradlew check' which is not to >>> far away from full build. Probably no one would expect a full build to fail >>> here, which again makes me think we need an equivalent Job on Jenkins here? >>> >>> Of course it is also fully reasonable to not support this full build in >>> line with that mini-monorepo. >>> >>> On the other hand, after filling the build cache - and enable more task >>> for caching - a full build should not be to expensive in general. Although >>> I d prefer some more granularity like e.g. 'gradlew -p sdks/java build' to >>> build all java, which, if I understand correctly, would be a side effect of >>> fixing [4]. >>> >>> [1] >>> https://github.com/apache/beam/blob/master/release/src/main/scripts/verify_release_build.sh#L142 >>> [2] >>> https://github.com/apache/beam/blob/master/release/src/main/scripts/verify_release_build.sh#L168-L190 >>> [3] https://beam.apache.org/contribute/ >>> [4] https://issues.apache.org/jira/browse/BEAM-4046 >>> >>> >>>> >>>> On Sun, Mar 10, 2019 at 7:05 PM Michael Luckey <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> while fiddling with beams release process and trying to get that >>>>> working on my machine, I again stumbled about 'gradlew build' >>>>> >>>>> Till now I am unable to get a full build run successfully on my >>>>> machine. Keeps failing. After setting up a clean docker instance and >>>>> trying >>>>> to run there, I also failed miserably. Which of kind is expected as part >>>>> of >>>>> the tests rely on docker itself which probably does not work out of the >>>>> box. But apart from these expected failures, there were also tons of other >>>>> failures. Need to setup a full blown vm and test again, but wanted to ask >>>>> here first as after looking into Jenkins setup, I was also unable to find >>>>> a >>>>> job which actually executes a full build. >>>>> >>>>> So I am wondering, whether >>>>> - anyone is actually able to successfully execute a full build >>>>> (preferable without build cache) >>>>> - it is intended we do only run a 'gradlew build' during release on >>>>> release managers box >>>>> - if that full build is somehow implicit in Jenkins setup, and I am >>>>> just not finding my way to the job/list of jobs which execute this full >>>>> build >>>>> >>>>> Any help/insights appreciated here. >>>>> >>>>> michel >>>>> >>>>> >>>>>
