Thanks everyone for the details. Seems like Java 11 support is farther along than I had imagined :)
> Is there any progress into getting > back, any ticket people can follow if interested? https://issues.apache.org/jira/browse/BEAM-10049 > I understand that a user can publish their own versions of HEAD containers but this does not work well when developing automated tests for distributed runners. Why not? On Wed, Jul 15, 2020 at 9:25 AM Chamikara Jayalath <[email protected]> wrote: > Can we consider regularly publishing HEAD containers as well (for example, > we publish SNAPSHOT jars daily) ? I understand that a user can publish > their own versions of HEAD containers but this does not work well when > developing automated tests for distributed runners. Apologies if this was > discussed before. > > Thanks, > Cham > > On Wed, Jul 15, 2020 at 12:43 AM Ismaël Mejía <[email protected]> wrote: > >> Thanks Robert for the explanation. Is there any progress into getting >> back, any ticket people can follow if interested? >> >> On Wed, Jul 15, 2020 at 12:13 AM Robert Burke <[email protected]> wrote: >> > >> > Disallowing the go containers was largely due to not having a simple >> check on the go boot code's licenses which is required for containers >> hosted under the main Apache namespace. >> > >> > A manual verification reveals it's only either Go's standard library >> BSD license and GRPCs Apache v2 licenses. Not impossible but not yet done >> by us. The JIRA issue has a link to the appropriate license finder for go >> packages. >> > >> > The amusing bit is that very similar Go boot code is included in the >> Java and Python containers too, so we're only accidentally in compliance >> with that there, if at all. >> > >> > >> > >> > On Tue, Jul 14, 2020, 2:22 PM Ismaël Mejía <[email protected]> wrote: >> >> >> >> +1 for naming as python containers, and quick release so users can try >> it. >> >> >> >> Not related to this tnread but I am also curious about the reasons to >> remove the >> >> go docker images, was this discussed/voted in the ML (maybe I missed >> it) ? >> >> >> >> I don't think Beam has been historically a conservative project about >> releasing >> >> early in-progress versions and I have learnt to appreciate this >> because it helps >> >> for early user testing and bug reports which will be definitely a must >> for Java >> >> 11. >> >> >> >> We should read the ticket Kyle mentions with a grain of salt. Most of >> the >> >> sub-tasks in that ticket are NOT about allowing users to run pipelines >> with Java >> >> 11 but about been able to fully build and run the tests and the source >> code >> >> ofBeam with Java 11 which is a different goal (important but probably >> less for >> >> end users) and a task with lots of extra issues because of plugins / >> dependent >> >> systems etc. >> >> >> >> For the Java 11 harness what we need is to guarantee is that users can >> run their >> >> code without issues with Java 11 and we can do this now for example by >> checking >> >> that portable runners that support Java 11 pass ValidatesRunner with >> the Java 11 >> >> harness. Since some classic runners [1] already pass these tests, it >> should be >> >> relatively 'easy' to do so for portable runners. >> >> >> >> [1] >> https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/ >> >> >> >> >> >> >> >> >> >> On Sat, Jul 11, 2020 at 12:43 AM Ahmet Altay <[email protected]> wrote: >> >> > >> >> > Related to the naming question, +1 and this will be similar to the >> python container naming (e.g. beam_python3.7_sdk). >> >> > >> >> > On Fri, Jul 10, 2020 at 1:46 PM Pablo Estrada <[email protected]> >> wrote: >> >> >> >> >> >> I agree with Kenn. Dataflow already has some publishing of >> non-portable JAva 11 containers, so I think it'll be great to formalize the >> process for portable containers, and let users play with it, and know of >> its availability. >> >> >> Best >> >> >> -P. >> >> >> >> >> >> On Fri, Jul 10, 2020 at 9:42 AM Kenneth Knowles <[email protected]> >> wrote: >> >> >>> >> >> >>> To the initial question: I'm +1 on the rename. The container is >> primarily something that the SDK should insert into the pipeline proto >> during construction, and only user-facing in more specialized situations. >> Given the state of Java and portability, it is a good time to get things >> named properly and unambiguously. I think a brief announce to dev@ and >> user@ when it happens is nice-to-have, but no need to give advance >> warning. >> >> >>> >> >> >>> Kenn >> >> >>> >> >> >>> On Fri, Jul 10, 2020 at 7:58 AM Kenneth Knowles <[email protected]> >> wrote: >> >> >>>> >> >> >>>> I believe Beam already has quite a few users that have forged >> ahead and used Java 11 with various runners, pre-portability. Mostly I >> believe the Java 11 limitations are with particular features (Schema >> codegen) and extensions/IOs/transitive deps. >> >> >>>> >> >> >>>> When it comes to the container, I'd be interested in looking at >> test coverage. The Flink & Spark portable ValidatesRunner suites use >> EMBEDDED environment, so they don't exercise the container. The first >> testing of the Java SDK harness container against the Python-based >> Universal Local Runner is in pull request now [1]. Are there other test >> suites to highlight? How hard would it be to run Flink & Spark against the >> container(s) too? >> >> >>>> >> >> >>>> Kenn >> >> >>>> >> >> >>>> [1] https://github.com/apache/beam/pull/11792 (despite the name >> ValidatesRunner, in this case it is validating both the runner and harness, >> since we don't have a compliance test suite for SDK harnesses) >> >> >>>> >> >> >>>> On Fri, Jul 10, 2020 at 7:54 AM Tyson Hamilton < >> [email protected]> wrote: >> >> >>>>> >> >> >>>>> What do we consider 'ready'? >> >> >>>>> >> >> >>>>> Maybe the only required outstanding bugs are supporting the >> direct runner (BEAM-10085), core tests (BEAM-10081), IO tests (BEAM-10084) >> to start with? Notably this would exclude failing tests like those for GCP >> core, GCPIOs, Dataflow runner, Spark runner, Flink runner, Samza. >> >> >>>>> >> >> >>>>> >> >> >>>>> On Thu, Jul 9, 2020 at 4:44 PM Kyle Weaver <[email protected]> >> wrote: >> >> >>>>>> >> >> >>>>>> My main question is, are we confident the Java 11 container is >> ready to release? AFAIK there are still a number of issues blocking full >> Java 11 support (cf [1]; not sure how many of these, if any, affect the SDK >> harness specifically though.) >> >> >>>>>> >> >> >>>>>> For comparison, we recently decided to stop publishing Go SDK >> containers until the Go SDK is considered mature [2]. In the meantime, >> those who want to use the Go SDK can build their own container images from >> source. >> >> >>>>>> >> >> >>>>>> Do we already have a Gradle task to build Java 11 containers? >> If not, this would be a good intermediate step, letting users opt-in to >> Java 11 without us overpromising support. >> >> >>>>> >> >> >>>>> >> >> >>>>> We do not. From what I can tell, the build.gradele [1] for the >> Java container is only for the one version. There is a docker file used for >> Jenkins tests. >> >> >>>>> >> >> >>>>> [1] >> https://github.com/apache/beam/blob/master/sdks/java/container/build.gradle >> >> >>>>> >> >> >>>>>> >> >> >>>>>> >> >> >>>>>> When we eventually do the renaming, we can add a note to >> CHANGES.md [3]. >> >> >>>>>> >> >> >>>>>> [1] https://issues.apache.org/jira/browse/BEAM-10090 >> >> >>>>>> [2] https://issues.apache.org/jira/browse/BEAM-9685 >> >> >>>>>> [3] https://github.com/apache/beam/blob/master/CHANGES.md >> >> >>>>>> >> >> >>>>>> On Thu, Jul 9, 2020 at 3:44 PM Emily Ye <[email protected]> >> wrote: >> >> >>>>>>> >> >> >>>>>>> Hi all, >> >> >>>>>>> >> >> >>>>>>> I'm getting ramped up on contributing and was looking into >> adding the Java 11 harness container to releases ( >> https://issues.apache.org/jira/browse/BEAM-8106) - should I rename the >> current java container so we have two new images `beam_java8_sdk` and >> `beam_java11_sdk` or hold off on renaming? If we do rename it, what steps >> should I take to announce/document the change? >> >> >>>>>>> >> >> >>>>>>> Thanks, >> >> >>>>>>> Emily >> >
