+1 to adding Jenkins jobs that test with Java 11. Might it also be possible to add a second java precommit that runs a subset of the tests on Java 11?
Andrew On Thu, Oct 18, 2018 at 8:52 AM Scott Wegner <sc...@apache.org> wrote: > If we just want to validate our produced artifacts work with Java11, I > believe it would be easy to start running some of our Jenkins jobs on Java > 11. beam_PostRelease_NightlyCandidate [1] is a good candidate: it runs the > quickstart examples across all runners from the SNAPSHOT published maven > architype [2]. > > [1] https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/ > [2] > https://github.com/apache/beam/blob/master/release/src/main/groovy/QuickstartArchetype.groovy > > On Thu, Oct 18, 2018 at 1:26 AM Ismaël Mejía <ieme...@gmail.com> wrote: > >> Yes, Kenn I was talking about building artifacts (because this is the >> strictest way to validate the compatiblity of our code base and find issues >> with some of the classpath/code generation we use or with the >> dependencies). I agree 100% with you, the real deal is to validate that we >> can use Java 8 built jars with more recent Java versions without issues and >> examples could be a good target for this. >> >> Supporting all non LTS java versions seems like a lot of work for a small >> return: (1) most users will probably stay in LTS versions, (2) not any >> single execution system has talked about supporting intermediary versions >> (some not even LTS) (3) this will augment the combination of tests we >> should execute to validate a release. Of course if we do care only about >> direct runner this could make sense. >> >> On Thu, Oct 18, 2018 at 9:30 AM Robert Bradshaw <rober...@google.com> >> wrote: >> >>> On Thu, Oct 18, 2018 at 4:55 AM Kenneth Knowles <k...@apache.org> wrote: >>> >>>> Just to add to what Luke said: The reason we had those Java 8-only >>>> modules was because some underlying tech (example: Gearpump) required Java >>>> 8. If an engine requires something then it is OK for a user who chooses the >>>> runner for that engine to also be subject to that requirement. Otherwise I >>>> would push back a bit on specialized modules. >>>> >>> >>> I, too, would avoid getting back into the situation where we have >>> different modules with different build/runtime requirements. (Java 8 was >>> special both because of Gearpump and because it introduced features such as >>> lambdas, type inference, and other functional libraries that were >>> particularly well suited to our programming model). >>> >>> >>>> Ismaël: Do we need to be thinking in terms of building artifacts? I >>>> think the important thing is if a user has a project and just pulls our >>>> jars from maven central that they can use JRE 11 to run it. I think what we >>>> need is a build matrix of user pipelines compiled and run w/ Java 8, 9, 10, >>>> 11 (or some subset). This can be obtained via the examples maven archetype, >>>> for example, having no relation at all to our own build tooling. I know >>>> Jenkins has a plugin for this, so we don't run into the "mega-Jenkins-job" >>>> problem. >>>> >>> >>> +1 for rephrasing it this way. In general one would expect our jars of >>> classfiles to work with later JREs. The magic we do with annotations and >>> bytecode generation may, however, require more support and testing. >>> >>> >>>> On Wed, Oct 17, 2018 at 2:03 PM Lukasz Cwik <lc...@google.com> wrote: >>>> >>>>> When we were supporting Java 7, there were Java 8 based modules in >>>>> Maven. We can follow a similar pattern where Java11 (or any other version) >>>>> have their own projects and as the base Java version moves up, we can >>>>> merge >>>>> those modules back in to simplify the set of modules being developed. >>>>> >>>>> On Wed, Oct 17, 2018 at 8:14 AM Ismaël Mejía <ieme...@gmail.com> >>>>> wrote: >>>>> >>>>>> I want to give more context on the previous work towards Java 11 >>>>>> support. The issue was not necessarily that gradle was not ok to support >>>>>> Java 11, but this is also a good point Lukasz. >>>>>> >>>>>> Months ago, we built a maven profile that basically set up Java to be >>>>>> strictly compatible with the ‘minimal’ java.base profile and used Java >>>>>> version (9/10) at the time. We found and fixed some missing dependencies >>>>>> (because Java 11 removed some of the Java EE stuff from the base >>>>>> profile), >>>>>> not sure if these changes were migrated to the gradle build. >>>>>> >>>>>> We also dealt with a good chunk of updates at the time of >>>>>> plugins/bytebuddy and other deps to guarantee that the execution was done >>>>>> strictly with the latest version of Java and its generated bytecode (the >>>>>> origin of the docker build images was to guarantee this isolation). At >>>>>> the >>>>>> time I run the tests of most IOs and most extensions with the existing >>>>>> (java 8 compiled) direct runner in JRE 10 with success but of course this >>>>>> was not the ultimate goal but a good way to evaluate the state of the >>>>>> code >>>>>> base (I remember some GCP tests were broken and the ApiSurfaceTests too). >>>>>> >>>>>> One important thing to remember is that the goal on this work was >>>>>> ‘limited’ to guarantee that a user that had ONLY Java 11 installed could >>>>>> build and generate Beam artifacts and run Beam programs in the direct >>>>>> runner. The limited scope was in part because of some issues of a full >>>>>> migration to Java 11: >>>>>> >>>>>> - Support for Java modules, this is a too much radical change and >>>>>> probably far from the scope of Beam at the moment. >>>>>> - Support for other runners, given that most execution systems do not >>>>>> support yet Java 11 this could imply a lot of extra work. In the >>>>>> meantime, >>>>>> this could be easier now due to the portability work, anyway, still not >>>>>> sure that Hadoop/Spark/Flink will be supporting Java 11 anytime soon. >>>>>> >>>>>> Let’s not forget that we still are and probably will be for a long >>>>>> time backwards compatible with Java 8 so we need to ensure that Java 11 >>>>>> only things don’t get into the code base. >>>>>> >>>>>> If you Lukasz or someone else is up to take this task, I think the >>>>>> immediate thing to do is to get back to the same state we were before the >>>>>> move to gradle, build the equivalent of the maven profile for the gradle >>>>>> build and tackle code generation and classpath issues, then try to fix >>>>>> missing parts and push this into the CI too (similar to the recent >>>>>> python 3 >>>>>> work) to guarantee that new changes don’t break Java 11 compatibility. I >>>>>> sadly lack of bandwidth to lead this task but I will gladly help with >>>>>> reviews, or feedback if someone else is up to the task. >>>>>> >>>>>> Not sure about more recent details, but probably worth evaluating at >>>>>> some moment is if we there is some interest on supporting multi-release >>>>>> jars with Java 11 classes or the dream scenario of having a AOT build of >>>>>> the direct runner. >>>>>> >>>>>> On Wed, Oct 17, 2018 at 12:06 PM Łukasz Gajowy < >>>>>> lukasz.gaj...@gmail.com> wrote: >>>>>> >>>>>>> FYI: If I'm correct, migrating to Java 11 may require Migrating to >>>>>>> Gradle 5, which to me is ok, but there are some issues on the way (such >>>>>>> as >>>>>>> [1] or maybe more that I'm not aware of). >>>>>>> >>>>>>> I did some research and it seems that Groovy 2.4.15 (the default >>>>>>> version of Groovy for Gradle 4.10.2) is unable to work with Java 11 due >>>>>>> to >>>>>>> Nest based access control problem [2]. The very same error happened >>>>>>> when I >>>>>>> tried to compile "core" module with java11. This issue is solved in >>>>>>> Groovy >>>>>>> 2.5.3 [3]. I was unable to run Gradle 4.10.2 with Groovy 2.5.3 >>>>>>> (NoClassDefError) and found out that the support for Groovy 2.5.3 will >>>>>>> be >>>>>>> added in Gradle 5.0RC1 (milestone) [4]. >>>>>>> >>>>>>> Other than that I had some issues with spottless and findbugs (I >>>>>>> turned them off for the sake of my experiment). >>>>>>> >>>>>>> Łukasz >>>>>>> >>>>>>> [1] https://issues.apache.org/jira/browse/BEAM-4042 >>>>>>> [2] >>>>>>> https://github.com/gradle/gradle/issues/5120#issuecomment-424610114 >>>>>>> [3] https://issues.apache.org/jira/browse/GROOVY-8727 >>>>>>> <https://www.google.com/url?q=https://issues.apache.org/jira/browse/GROOVY-8727&sa=D&ust=1539701479825000&usg=AFQjCNGwndnRaDx9NAr0cJ-isZpRWngcjw> >>>>>>> [4] https://github.com/gradle/gradle/pull/7376 >>>>>>> <https://www.google.com/url?q=https://github.com/gradle/gradle/pull/7376&sa=D&ust=1539701479825000&usg=AFQjCNE9DOokgVxojvsvjsaJ8_6KY2M92Q> >>>>>>> >>>>>>> >>>>>>> czw., 11 paź 2018 o 13:10 Łukasz Gajowy <lukasz.gaj...@gmail.com> >>>>>>> napisał(a): >>>>>>> >>>>>>>> FWIW, the issue regarding java 11 support to gradle might be >>>>>>>> helpful here [1]. Quoting: "there are only some minor corner cases that >>>>>>>> don't work. Overall Java 11 support is pretty solid already" but some >>>>>>>> users >>>>>>>> reported problems in comments with Checkstyle plugin and MacOS + >>>>>>>> Gradle4.10.2 (this might be important for us). >>>>>>>> >>>>>>>> Łukasz >>>>>>>> >>>>>>>> [1] https://github.com/gradle/gradle/issues/5120 >>>>>>>> >>>>>>>> czw., 11 paź 2018 o 02:06 Pablo Estrada <pabl...@google.com> >>>>>>>> napisał(a): >>>>>>>> >>>>>>>>> Hello all, >>>>>>>>> If I understand you correctly Ismael, a good amount of >>>>>>>>> 'beam-sdks-java-core' tests are already passing with Java 11, so the >>>>>>>>> amount >>>>>>>>> of work necessary on the core module should be relatively small. Is >>>>>>>>> this >>>>>>>>> correct? Are there improvements that may be missing in terms of >>>>>>>>> modularization? >>>>>>>>> >>>>>>>>> There is also the work necessary to build/run tests with Gradle.... >>>>>>>>> >>>>>>>>> I am also curious... how much work do you estimate is necessary to >>>>>>>>> support Java 11 with some of the existing sources? I understand that >>>>>>>>> we >>>>>>>>> have many, many sources, but perhaps some of the more popular ones >>>>>>>>> (e.g. >>>>>>>>> TextIO)? >>>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> -P. >>>>>>>>> >>>>>>>>> On Wed, Oct 10, 2018 at 12:59 AM Arif Kasim <arifka...@google.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Thanks for the clarification Ismaël. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> * • **Arif Kasim* >>>>>>>>>> * • * Strategic Cloud Engineer >>>>>>>>>> * • *Google, Inc. >>>>>>>>>> • arifka...@google.com >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Oct 10, 2018 at 9:41 AM Ismaël Mejía <ieme...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Just wanted to clarify, there is already a JIRA for ongoing work >>>>>>>>>>> on >>>>>>>>>>> Java 11 support. >>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-2530 >>>>>>>>>>> >>>>>>>>>>> I led the initial work on supporting what at the time was Java >>>>>>>>>>> 9/10, >>>>>>>>>>> so far the biggest blockers were around the ApiSurface tests >>>>>>>>>>> (not at >>>>>>>>>>> all compatible with these versions) but at the time we were at 5 >>>>>>>>>>> tests >>>>>>>>>>> from getting sdks/core passing. Notice also that the scope of >>>>>>>>>>> this >>>>>>>>>>> JIRA evolved to support only the LTS version (Java 11), and >>>>>>>>>>> specifically to support only sdks/core + direct runner. >>>>>>>>>>> Supporting all >>>>>>>>>>> IOs or runners really is more a question of the dependencies >>>>>>>>>>> working >>>>>>>>>>> nicely with Java 11 so this will probably take long time. Also >>>>>>>>>>> the >>>>>>>>>>> idea so far does NOT include supporting the Java module system >>>>>>>>>>> at all. >>>>>>>>>>> >>>>>>>>>>> I stopped working on this during the move to gradle because it >>>>>>>>>>> was too >>>>>>>>>>> hard to tackle both Java evolving and all the ongoing changes in >>>>>>>>>>> the >>>>>>>>>>> build system. If somebody in the community wants to contribute >>>>>>>>>>> in this >>>>>>>>>>> area it will be greatly appreciated, notice that all the work we >>>>>>>>>>> did >>>>>>>>>>> on the build system for this needs to be implemented now in >>>>>>>>>>> gradle >>>>>>>>>>> too. >>>>>>>>>>> On Sat, Oct 6, 2018 at 5:55 PM Romain Manni-Bucau < >>>>>>>>>>> rmannibu...@gmail.com> wrote: >>>>>>>>>>> > >>>>>>>>>>> > @Reuven: bytebuddy by itself no but the way beam tries to >>>>>>>>>>> inject the proxy class is. There are other strategies you can use in >>>>>>>>>>> bytebuddy which work. >>>>>>>>>>> > >>>>>>>>>>> > Romain Manni-Bucau >>>>>>>>>>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > Le sam. 6 oct. 2018 à 17:51, Reuven Lax <re...@google.com> a >>>>>>>>>>> écrit : >>>>>>>>>>> >> >>>>>>>>>>> >> Romain, do you have any more details on the ByteBuddy >>>>>>>>>>> incompatibility? Is ByteBuddy incompatible with the Java 11 JRE, or >>>>>>>>>>> just >>>>>>>>>>> with new language features? >>>>>>>>>>> >> >>>>>>>>>>> >> On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau < >>>>>>>>>>> rmannibu...@gmail.com> wrote: >>>>>>>>>>> >>> >>>>>>>>>>> >>> Hi Arif, >>>>>>>>>>> >>> >>>>>>>>>>> >>> AFAIK bytebuddy code is not java 11 friendly otherwise it >>>>>>>>>>> runs (but it means your pipeline is very very simple since it does >>>>>>>>>>> not have >>>>>>>>>>> a dofn ;)) if your engine supports it. Also note that the modules >>>>>>>>>>> not being >>>>>>>>>>> named you can have to use some weird import names or even unstable >>>>>>>>>>> ones if >>>>>>>>>>> you want to use modules (but there is no real reason to do that yet >>>>>>>>>>> in >>>>>>>>>>> java). >>>>>>>>>>> >>> >>>>>>>>>>> >>> Romain Manni-Bucau >>>>>>>>>>> >>> @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>>>>>>>>> >>> >>>>>>>>>>> >>> >>>>>>>>>>> >>> Le ven. 5 oct. 2018 à 19:10, Arif Kasim < >>>>>>>>>>> arifka...@google.com> a écrit : >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> Hello, >>>>>>>>>>> >>>> What's the status of java version > 8 support for beam? >>>>>>>>>>> Thanks. >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> -Arif. >>>>>>>>>>> >>>>>>>>>> > > -- > > > > > Got feedback? tinyurl.com/swegner-feedback >
