Disallowing the go containers was largely due to not having a simple check on the go boot code's licenses which is required for containers hosted under the main Apache namespace.
A manual verification reveals it's only either Go's standard library BSD license and GRPCs Apache v2 licenses. Not impossible but not yet done by us. The JIRA issue has a link to the appropriate license finder for go packages. The amusing bit is that very similar Go boot code is included in the Java and Python containers too, so we're only accidentally in compliance with that there, if at all. On Tue, Jul 14, 2020, 2:22 PM Ismaël Mejía <[email protected]> wrote: > +1 for naming as python containers, and quick release so users can try it. > > Not related to this tnread but I am also curious about the reasons to > remove the > go docker images, was this discussed/voted in the ML (maybe I missed it) ? > > I don't think Beam has been historically a conservative project about > releasing > early in-progress versions and I have learnt to appreciate this because it > helps > for early user testing and bug reports which will be definitely a must for > Java > 11. > > We should read the ticket Kyle mentions with a grain of salt. Most of the > sub-tasks in that ticket are NOT about allowing users to run pipelines > with Java > 11 but about been able to fully build and run the tests and the source code > ofBeam with Java 11 which is a different goal (important but probably less > for > end users) and a task with lots of extra issues because of plugins / > dependent > systems etc. > > For the Java 11 harness what we need is to guarantee is that users can run > their > code without issues with Java 11 and we can do this now for example by > checking > that portable runners that support Java 11 pass ValidatesRunner with the > Java 11 > harness. Since some classic runners [1] already pass these tests, it > should be > relatively 'easy' to do so for portable runners. > > [1] > https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/ > > > > > On Sat, Jul 11, 2020 at 12:43 AM Ahmet Altay <[email protected]> wrote: > > > > Related to the naming question, +1 and this will be similar to the > python container naming (e.g. beam_python3.7_sdk). > > > > On Fri, Jul 10, 2020 at 1:46 PM Pablo Estrada <[email protected]> > wrote: > >> > >> I agree with Kenn. Dataflow already has some publishing of non-portable > JAva 11 containers, so I think it'll be great to formalize the process for > portable containers, and let users play with it, and know of its > availability. > >> Best > >> -P. > >> > >> On Fri, Jul 10, 2020 at 9:42 AM Kenneth Knowles <[email protected]> > wrote: > >>> > >>> To the initial question: I'm +1 on the rename. The container is > primarily something that the SDK should insert into the pipeline proto > during construction, and only user-facing in more specialized situations. > Given the state of Java and portability, it is a good time to get things > named properly and unambiguously. I think a brief announce to dev@ and > user@ when it happens is nice-to-have, but no need to give advance > warning. > >>> > >>> Kenn > >>> > >>> On Fri, Jul 10, 2020 at 7:58 AM Kenneth Knowles <[email protected]> > wrote: > >>>> > >>>> I believe Beam already has quite a few users that have forged ahead > and used Java 11 with various runners, pre-portability. Mostly I believe > the Java 11 limitations are with particular features (Schema codegen) and > extensions/IOs/transitive deps. > >>>> > >>>> When it comes to the container, I'd be interested in looking at test > coverage. The Flink & Spark portable ValidatesRunner suites use EMBEDDED > environment, so they don't exercise the container. The first testing of the > Java SDK harness container against the Python-based Universal Local Runner > is in pull request now [1]. Are there other test suites to highlight? How > hard would it be to run Flink & Spark against the container(s) too? > >>>> > >>>> Kenn > >>>> > >>>> [1] https://github.com/apache/beam/pull/11792 (despite the name > ValidatesRunner, in this case it is validating both the runner and harness, > since we don't have a compliance test suite for SDK harnesses) > >>>> > >>>> On Fri, Jul 10, 2020 at 7:54 AM Tyson Hamilton <[email protected]> > wrote: > >>>>> > >>>>> What do we consider 'ready'? > >>>>> > >>>>> Maybe the only required outstanding bugs are supporting the direct > runner (BEAM-10085), core tests (BEAM-10081), IO tests (BEAM-10084) to > start with? Notably this would exclude failing tests like those for GCP > core, GCPIOs, Dataflow runner, Spark runner, Flink runner, Samza. > >>>>> > >>>>> > >>>>> On Thu, Jul 9, 2020 at 4:44 PM Kyle Weaver <[email protected]> > wrote: > >>>>>> > >>>>>> My main question is, are we confident the Java 11 container is > ready to release? AFAIK there are still a number of issues blocking full > Java 11 support (cf [1]; not sure how many of these, if any, affect the SDK > harness specifically though.) > >>>>>> > >>>>>> For comparison, we recently decided to stop publishing Go SDK > containers until the Go SDK is considered mature [2]. In the meantime, > those who want to use the Go SDK can build their own container images from > source. > >>>>>> > >>>>>> Do we already have a Gradle task to build Java 11 containers? If > not, this would be a good intermediate step, letting users opt-in to Java > 11 without us overpromising support. > >>>>> > >>>>> > >>>>> We do not. From what I can tell, the build.gradele [1] for the Java > container is only for the one version. There is a docker file used for > Jenkins tests. > >>>>> > >>>>> [1] > https://github.com/apache/beam/blob/master/sdks/java/container/build.gradle > >>>>> > >>>>>> > >>>>>> > >>>>>> When we eventually do the renaming, we can add a note to CHANGES.md > [3]. > >>>>>> > >>>>>> [1] https://issues.apache.org/jira/browse/BEAM-10090 > >>>>>> [2] https://issues.apache.org/jira/browse/BEAM-9685 > >>>>>> [3] https://github.com/apache/beam/blob/master/CHANGES.md > >>>>>> > >>>>>> On Thu, Jul 9, 2020 at 3:44 PM Emily Ye <[email protected]> wrote: > >>>>>>> > >>>>>>> Hi all, > >>>>>>> > >>>>>>> I'm getting ramped up on contributing and was looking into adding > the Java 11 harness container to releases ( > https://issues.apache.org/jira/browse/BEAM-8106) - should I rename the > current java container so we have two new images `beam_java8_sdk` and > `beam_java11_sdk` or hold off on renaming? If we do rename it, what steps > should I take to announce/document the change? > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Emily >
