If we just want to validate our produced artifacts work with Java11, I believe it would be easy to start running some of our Jenkins jobs on Java 11. beam_PostRelease_NightlyCandidate [1] is a good candidate: it runs the quickstart examples across all runners from the SNAPSHOT published maven architype [2].
[1] https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/ [2] https://github.com/apache/beam/blob/master/release/src/main/groovy/QuickstartArchetype.groovy On Thu, Oct 18, 2018 at 1:26 AM Ismaël Mejía <ieme...@gmail.com> wrote: > Yes, Kenn I was talking about building artifacts (because this is the > strictest way to validate the compatiblity of our code base and find issues > with some of the classpath/code generation we use or with the > dependencies). I agree 100% with you, the real deal is to validate that we > can use Java 8 built jars with more recent Java versions without issues and > examples could be a good target for this. > > Supporting all non LTS java versions seems like a lot of work for a small > return: (1) most users will probably stay in LTS versions, (2) not any > single execution system has talked about supporting intermediary versions > (some not even LTS) (3) this will augment the combination of tests we > should execute to validate a release. Of course if we do care only about > direct runner this could make sense. > > On Thu, Oct 18, 2018 at 9:30 AM Robert Bradshaw <rober...@google.com> > wrote: > >> On Thu, Oct 18, 2018 at 4:55 AM Kenneth Knowles <k...@apache.org> wrote: >> >>> Just to add to what Luke said: The reason we had those Java 8-only >>> modules was because some underlying tech (example: Gearpump) required Java >>> 8. If an engine requires something then it is OK for a user who chooses the >>> runner for that engine to also be subject to that requirement. Otherwise I >>> would push back a bit on specialized modules. >>> >> >> I, too, would avoid getting back into the situation where we have >> different modules with different build/runtime requirements. (Java 8 was >> special both because of Gearpump and because it introduced features such as >> lambdas, type inference, and other functional libraries that were >> particularly well suited to our programming model). >> >> >>> Ismaël: Do we need to be thinking in terms of building artifacts? I >>> think the important thing is if a user has a project and just pulls our >>> jars from maven central that they can use JRE 11 to run it. I think what we >>> need is a build matrix of user pipelines compiled and run w/ Java 8, 9, 10, >>> 11 (or some subset). This can be obtained via the examples maven archetype, >>> for example, having no relation at all to our own build tooling. I know >>> Jenkins has a plugin for this, so we don't run into the "mega-Jenkins-job" >>> problem. >>> >> >> +1 for rephrasing it this way. In general one would expect our jars of >> classfiles to work with later JREs. The magic we do with annotations and >> bytecode generation may, however, require more support and testing. >> >> >>> On Wed, Oct 17, 2018 at 2:03 PM Lukasz Cwik <lc...@google.com> wrote: >>> >>>> When we were supporting Java 7, there were Java 8 based modules in >>>> Maven. We can follow a similar pattern where Java11 (or any other version) >>>> have their own projects and as the base Java version moves up, we can merge >>>> those modules back in to simplify the set of modules being developed. >>>> >>>> On Wed, Oct 17, 2018 at 8:14 AM Ismaël Mejía <ieme...@gmail.com> wrote: >>>> >>>>> I want to give more context on the previous work towards Java 11 >>>>> support. The issue was not necessarily that gradle was not ok to support >>>>> Java 11, but this is also a good point Lukasz. >>>>> >>>>> Months ago, we built a maven profile that basically set up Java to be >>>>> strictly compatible with the ‘minimal’ java.base profile and used Java >>>>> version (9/10) at the time. We found and fixed some missing dependencies >>>>> (because Java 11 removed some of the Java EE stuff from the base profile), >>>>> not sure if these changes were migrated to the gradle build. >>>>> >>>>> We also dealt with a good chunk of updates at the time of >>>>> plugins/bytebuddy and other deps to guarantee that the execution was done >>>>> strictly with the latest version of Java and its generated bytecode (the >>>>> origin of the docker build images was to guarantee this isolation). At the >>>>> time I run the tests of most IOs and most extensions with the existing >>>>> (java 8 compiled) direct runner in JRE 10 with success but of course this >>>>> was not the ultimate goal but a good way to evaluate the state of the code >>>>> base (I remember some GCP tests were broken and the ApiSurfaceTests too). >>>>> >>>>> One important thing to remember is that the goal on this work was >>>>> ‘limited’ to guarantee that a user that had ONLY Java 11 installed could >>>>> build and generate Beam artifacts and run Beam programs in the direct >>>>> runner. The limited scope was in part because of some issues of a full >>>>> migration to Java 11: >>>>> >>>>> - Support for Java modules, this is a too much radical change and >>>>> probably far from the scope of Beam at the moment. >>>>> - Support for other runners, given that most execution systems do not >>>>> support yet Java 11 this could imply a lot of extra work. In the meantime, >>>>> this could be easier now due to the portability work, anyway, still not >>>>> sure that Hadoop/Spark/Flink will be supporting Java 11 anytime soon. >>>>> >>>>> Let’s not forget that we still are and probably will be for a long >>>>> time backwards compatible with Java 8 so we need to ensure that Java 11 >>>>> only things don’t get into the code base. >>>>> >>>>> If you Lukasz or someone else is up to take this task, I think the >>>>> immediate thing to do is to get back to the same state we were before the >>>>> move to gradle, build the equivalent of the maven profile for the gradle >>>>> build and tackle code generation and classpath issues, then try to fix >>>>> missing parts and push this into the CI too (similar to the recent python >>>>> 3 >>>>> work) to guarantee that new changes don’t break Java 11 compatibility. I >>>>> sadly lack of bandwidth to lead this task but I will gladly help with >>>>> reviews, or feedback if someone else is up to the task. >>>>> >>>>> Not sure about more recent details, but probably worth evaluating at >>>>> some moment is if we there is some interest on supporting multi-release >>>>> jars with Java 11 classes or the dream scenario of having a AOT build of >>>>> the direct runner. >>>>> >>>>> On Wed, Oct 17, 2018 at 12:06 PM Łukasz Gajowy < >>>>> lukasz.gaj...@gmail.com> wrote: >>>>> >>>>>> FYI: If I'm correct, migrating to Java 11 may require Migrating to >>>>>> Gradle 5, which to me is ok, but there are some issues on the way (such >>>>>> as >>>>>> [1] or maybe more that I'm not aware of). >>>>>> >>>>>> I did some research and it seems that Groovy 2.4.15 (the default >>>>>> version of Groovy for Gradle 4.10.2) is unable to work with Java 11 due >>>>>> to >>>>>> Nest based access control problem [2]. The very same error happened when >>>>>> I >>>>>> tried to compile "core" module with java11. This issue is solved in >>>>>> Groovy >>>>>> 2.5.3 [3]. I was unable to run Gradle 4.10.2 with Groovy 2.5.3 >>>>>> (NoClassDefError) and found out that the support for Groovy 2.5.3 will be >>>>>> added in Gradle 5.0RC1 (milestone) [4]. >>>>>> >>>>>> Other than that I had some issues with spottless and findbugs (I >>>>>> turned them off for the sake of my experiment). >>>>>> >>>>>> Łukasz >>>>>> >>>>>> [1] https://issues.apache.org/jira/browse/BEAM-4042 >>>>>> [2] >>>>>> https://github.com/gradle/gradle/issues/5120#issuecomment-424610114 >>>>>> [3] https://issues.apache.org/jira/browse/GROOVY-8727 >>>>>> <https://www.google.com/url?q=https://issues.apache.org/jira/browse/GROOVY-8727&sa=D&ust=1539701479825000&usg=AFQjCNGwndnRaDx9NAr0cJ-isZpRWngcjw> >>>>>> [4] https://github.com/gradle/gradle/pull/7376 >>>>>> <https://www.google.com/url?q=https://github.com/gradle/gradle/pull/7376&sa=D&ust=1539701479825000&usg=AFQjCNE9DOokgVxojvsvjsaJ8_6KY2M92Q> >>>>>> >>>>>> >>>>>> czw., 11 paź 2018 o 13:10 Łukasz Gajowy <lukasz.gaj...@gmail.com> >>>>>> napisał(a): >>>>>> >>>>>>> FWIW, the issue regarding java 11 support to gradle might be helpful >>>>>>> here [1]. Quoting: "there are only some minor corner cases that don't >>>>>>> work. >>>>>>> Overall Java 11 support is pretty solid already" but some users reported >>>>>>> problems in comments with Checkstyle plugin and MacOS + Gradle4.10.2 >>>>>>> (this >>>>>>> might be important for us). >>>>>>> >>>>>>> Łukasz >>>>>>> >>>>>>> [1] https://github.com/gradle/gradle/issues/5120 >>>>>>> >>>>>>> czw., 11 paź 2018 o 02:06 Pablo Estrada <pabl...@google.com> >>>>>>> napisał(a): >>>>>>> >>>>>>>> Hello all, >>>>>>>> If I understand you correctly Ismael, a good amount of >>>>>>>> 'beam-sdks-java-core' tests are already passing with Java 11, so the >>>>>>>> amount >>>>>>>> of work necessary on the core module should be relatively small. Is >>>>>>>> this >>>>>>>> correct? Are there improvements that may be missing in terms of >>>>>>>> modularization? >>>>>>>> >>>>>>>> There is also the work necessary to build/run tests with Gradle.... >>>>>>>> >>>>>>>> I am also curious... how much work do you estimate is necessary to >>>>>>>> support Java 11 with some of the existing sources? I understand that we >>>>>>>> have many, many sources, but perhaps some of the more popular ones >>>>>>>> (e.g. >>>>>>>> TextIO)? >>>>>>>> >>>>>>>> Thanks! >>>>>>>> -P. >>>>>>>> >>>>>>>> On Wed, Oct 10, 2018 at 12:59 AM Arif Kasim <arifka...@google.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thanks for the clarification Ismaël. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> * • **Arif Kasim* >>>>>>>>> * • * Strategic Cloud Engineer >>>>>>>>> * • *Google, Inc. >>>>>>>>> • arifka...@google.com >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Oct 10, 2018 at 9:41 AM Ismaël Mejía <ieme...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Just wanted to clarify, there is already a JIRA for ongoing work >>>>>>>>>> on >>>>>>>>>> Java 11 support. >>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-2530 >>>>>>>>>> >>>>>>>>>> I led the initial work on supporting what at the time was Java >>>>>>>>>> 9/10, >>>>>>>>>> so far the biggest blockers were around the ApiSurface tests (not >>>>>>>>>> at >>>>>>>>>> all compatible with these versions) but at the time we were at 5 >>>>>>>>>> tests >>>>>>>>>> from getting sdks/core passing. Notice also that the scope of this >>>>>>>>>> JIRA evolved to support only the LTS version (Java 11), and >>>>>>>>>> specifically to support only sdks/core + direct runner. >>>>>>>>>> Supporting all >>>>>>>>>> IOs or runners really is more a question of the dependencies >>>>>>>>>> working >>>>>>>>>> nicely with Java 11 so this will probably take long time. Also the >>>>>>>>>> idea so far does NOT include supporting the Java module system at >>>>>>>>>> all. >>>>>>>>>> >>>>>>>>>> I stopped working on this during the move to gradle because it >>>>>>>>>> was too >>>>>>>>>> hard to tackle both Java evolving and all the ongoing changes in >>>>>>>>>> the >>>>>>>>>> build system. If somebody in the community wants to contribute in >>>>>>>>>> this >>>>>>>>>> area it will be greatly appreciated, notice that all the work we >>>>>>>>>> did >>>>>>>>>> on the build system for this needs to be implemented now in gradle >>>>>>>>>> too. >>>>>>>>>> On Sat, Oct 6, 2018 at 5:55 PM Romain Manni-Bucau < >>>>>>>>>> rmannibu...@gmail.com> wrote: >>>>>>>>>> > >>>>>>>>>> > @Reuven: bytebuddy by itself no but the way beam tries to >>>>>>>>>> inject the proxy class is. There are other strategies you can use in >>>>>>>>>> bytebuddy which work. >>>>>>>>>> > >>>>>>>>>> > Romain Manni-Bucau >>>>>>>>>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > Le sam. 6 oct. 2018 à 17:51, Reuven Lax <re...@google.com> a >>>>>>>>>> écrit : >>>>>>>>>> >> >>>>>>>>>> >> Romain, do you have any more details on the ByteBuddy >>>>>>>>>> incompatibility? Is ByteBuddy incompatible with the Java 11 JRE, or >>>>>>>>>> just >>>>>>>>>> with new language features? >>>>>>>>>> >> >>>>>>>>>> >> On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau < >>>>>>>>>> rmannibu...@gmail.com> wrote: >>>>>>>>>> >>> >>>>>>>>>> >>> Hi Arif, >>>>>>>>>> >>> >>>>>>>>>> >>> AFAIK bytebuddy code is not java 11 friendly otherwise it >>>>>>>>>> runs (but it means your pipeline is very very simple since it does >>>>>>>>>> not have >>>>>>>>>> a dofn ;)) if your engine supports it. Also note that the modules >>>>>>>>>> not being >>>>>>>>>> named you can have to use some weird import names or even unstable >>>>>>>>>> ones if >>>>>>>>>> you want to use modules (but there is no real reason to do that yet >>>>>>>>>> in >>>>>>>>>> java). >>>>>>>>>> >>> >>>>>>>>>> >>> Romain Manni-Bucau >>>>>>>>>> >>> @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> >>> Le ven. 5 oct. 2018 à 19:10, Arif Kasim <arifka...@google.com> >>>>>>>>>> a écrit : >>>>>>>>>> >>>> >>>>>>>>>> >>>> Hello, >>>>>>>>>> >>>> What's the status of java version > 8 support for beam? >>>>>>>>>> Thanks. >>>>>>>>>> >>>> >>>>>>>>>> >>>> -Arif. >>>>>>>>>> >>>>>>>>> -- Got feedback? tinyurl.com/swegner-feedback
