I agree with Kenn. Dataflow already has some publishing of non-portable JAva 11 containers, so I think it'll be great to formalize the process for portable containers, and let users play with it, and know of its availability. Best -P.
On Fri, Jul 10, 2020 at 9:42 AM Kenneth Knowles <[email protected]> wrote: > To the initial question: I'm +1 on the rename. The container is primarily > something that the SDK should insert into the pipeline proto during > construction, and only user-facing in more specialized situations. Given > the state of Java and portability, it is a good time to get things named > properly and unambiguously. I think a brief announce to dev@ and user@ > when it happens is nice-to-have, but no need to give advance warning. > > Kenn > > On Fri, Jul 10, 2020 at 7:58 AM Kenneth Knowles <[email protected]> wrote: > >> I believe Beam already has quite a few users that have forged ahead and >> used Java 11 with various runners, pre-portability. Mostly I believe the >> Java 11 limitations are with particular features (Schema codegen) and >> extensions/IOs/transitive deps. >> >> When it comes to the container, I'd be interested in looking at test >> coverage. The Flink & Spark portable ValidatesRunner suites use EMBEDDED >> environment, so they don't exercise the container. The first testing of the >> Java SDK harness container against the Python-based Universal Local Runner >> is in pull request now [1]. Are there other test suites to highlight? How >> hard would it be to run Flink & Spark against the container(s) too? >> >> Kenn >> >> [1] https://github.com/apache/beam/pull/11792 (despite the name >> ValidatesRunner, in this case it is validating both the runner and harness, >> since we don't have a compliance test suite for SDK harnesses) >> >> On Fri, Jul 10, 2020 at 7:54 AM Tyson Hamilton <[email protected]> >> wrote: >> >>> What do we consider 'ready'? >>> >>> Maybe the only required outstanding bugs are supporting the direct >>> runner (BEAM-10085), core tests (BEAM-10081), IO tests (BEAM-10084) to >>> start with? Notably this would exclude failing tests like those for GCP >>> core, GCPIOs, Dataflow runner, Spark runner, Flink runner, Samza. >>> >>> >>> On Thu, Jul 9, 2020 at 4:44 PM Kyle Weaver <[email protected]> wrote: >>> >>>> My main question is, are we confident the Java 11 container is ready to >>>> release? AFAIK there are still a number of issues blocking full Java 11 >>>> support (cf [1] <https://issues.apache.org/jira/browse/BEAM-10090>; not >>>> sure how many of these, if any, affect the SDK harness specifically >>>> though.) >>>> >>>> For comparison, we recently decided to stop publishing Go SDK >>>> containers until the Go SDK is considered mature [2]. In the meantime, >>>> those who want to use the Go SDK can build their own container images from >>>> source. >>>> >>>> Do we already have a Gradle task to build Java 11 containers? If not, >>>> this would be a good intermediate step, letting users opt-in to Java >>>> 11 without us overpromising support. >>>> >>> >>> We do not. From what I can tell, the build.gradele [1] for the Java >>> container is only for the one version. There is a docker file used for >>> Jenkins tests. >>> >>> [1] >>> https://github.com/apache/beam/blob/master/sdks/java/container/build.gradle >>> >>> >>>> >>> When we eventually do the renaming, we can add a note to CHANGES.md [3]. >>>> >>>> [1] https://issues.apache.org/jira/browse/BEAM-10090 >>>> [2] https://issues.apache.org/jira/browse/BEAM-9685 >>>> [3] https://github.com/apache/beam/blob/master/CHANGES.md >>>> >>>> On Thu, Jul 9, 2020 at 3:44 PM Emily Ye <[email protected]> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I'm getting ramped up on contributing and was looking into adding the >>>>> Java 11 harness container to releases ( >>>>> https://issues.apache.org/jira/browse/BEAM-8106) - should I rename >>>>> the current java container so we have two new images `beam_java8_sdk` and >>>>> `beam_java11_sdk` or hold off on renaming? If we do rename it, what steps >>>>> should I take to announce/document the change? >>>>> >>>>> Thanks, >>>>> Emily >>>>> >>>>
