Just to give an update on this, I plan on creating an RC3 as soon as I get the dataflow containers rebuilt. (Been busy with the beam summit among other things.)
Except for direct runner fixes and the dataflow tags, I expect it to be the same as RC2, so testing on flink/apex/spark done now would be applicable to the next RC. https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475 . On Fri, Mar 9, 2018 at 4:24 PM Charles Chen <c...@google.com> wrote: > Thank you Valentyn for reporting this. I have traced the issue back to > https://github.com/apache/beam/pull/4666, so I have sent out a PR to fix: > https://github.com/apache/beam/pull/4846. > > On Fri, Mar 9, 2018 at 2:17 PM, Valentyn Tymofieiev <valen...@google.com> > wrote: > >> -1. >> >> Checked Python Quickstarts (Passed) and Python MobileGaming on >> DirectRunner. I observe an issue in BQ sink for hourly teams score example: >> >> https://issues.apache.org/jira/browse/BEAM-3824 >> >> On Fri, Mar 9, 2018 at 10:49 AM, Lukasz Cwik <lc...@google.com> wrote: >> >>> I checked that word count quickstarts (except Dataflow) worked for RC2 >>> to hopefully prevent an RC4. >>> >>> On Fri, Mar 9, 2018 at 10:29 AM, Robert Bradshaw <rober...@google.com> >>> wrote: >>> >>>> Thanks, Alan, for pointing this out. I see this now, and it looks like I >>>> need to finish building the dataflow workers so they have something to >>>> point to. I will do this and release an RC3 once that's ready. >>>> >>>> In the meantime, it'd be great if we could validate everything else >>>> about >>>> this RC such that when this on-line, dataflow-only change is out we >>>> won't >>>> have any further surprises. I see Luke went through the Java Quickstart >>>> examples, thanks! >>>> >>>> >>>> On Thu, Mar 8, 2018 at 3:48 PM Lukasz Cwik <lc...@google.com> wrote: >>>> >>>> > Yes, the release guide has a segment "Update release specific >>>> configurations" that has a tidbit about this. >>>> >>>> > On Thu, Mar 8, 2018 at 3:45 PM, Alan Myrvold <amyrv...@google.com> >>>> wrote: >>>> >>>> >> The dataflow java worker version wasn't updated on the branch as in >>>> past >>>> releases ... should it be? >>>> >> https://issues.apache.org/jira/browse/BEAM-3815 >>>> >>>> >>>> >> On Thu, Mar 8, 2018 at 1:40 PM Romain Manni-Bucau < >>>> rmannibu...@gmail.com> >>>> wrote: >>>> >>>> >>> Can still be provided as a generic one (like the an offset or key >>>> based >>>> one) but good enough for now, right, was just surprising to not see it >>>> when >>>> checking the breakage. >>>> >>>> >>> Le 8 mars 2018 22:05, "Eugene Kirpichov" <kirpic...@google.com> a >>>> écrit >>>> : >>>> >>>> >>> All SDF-related method annotations in DoFn are marked >>>> @Experimental. I >>>> guess that should apply to RestrictionTracker too, but I wouldn't be too >>>> worried about that, since it only makes sense to use in the context of >>>> those methods. >>>> >>>> >>> On Thu, Mar 8, 2018 at 12:36 PM Romain Manni-Bucau < >>>> rmannibu...@gmail.com> wrote: >>>> >>>> >>>> Hmm, does sdf api misses some @Experimental then? >>>> >>>> >>>> To clarify: for waitUntilFinish I'm fine with the 2.4 as this but >>>> cant >>>> +1 or +0 since none of my tests pass reliably in current state without a >>>> retry strategy making the call useless. >>>> >>>> >>>> Le 8 mars 2018 21:02, "Reuven Lax" <re...@google.com> a écrit : >>>> >>>> >>>>> Does Nexmark use SerializableCoder? >>>> >>>> >>>> >>>>> On Thu, Mar 8, 2018 at 10:42 AM Robert Bradshaw < >>>> rober...@google.com> >>>> wrote: >>>> >>>> >>>>>> I put the validation checklist spreadsheet is up at >>>> >>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475 >>>> >>>> >>>>>> Regarding the direct runner regression on query 10, this is >>>> understandable given how mutation detection has been changed for >>>> serializable coders (and should be tracked, probably fixed by avoiding >>>> SerializableCoder). It should not affect other runners. Could you file a >>>> bug? >>>> >>>> >>>>>> Regarding waitUntilFinish, this is a bug but not a blocker--it's >>>> been this way since teardown was introduced. There are many >>>> nice-to-haves >>>> that one could merge from master to the release branch, but we've seen >>>> where that trend leads. >>>> >>>> >>>>>> Regarding the backwards incompatible changes in restriction >>>> tracker, >>>> this is (as I understand it) a change to the experimental SDF API. >>>> Eugene, >>>> do you want to comment on this? >>>> >>>> >>>> >>>> >>>>>> On Thu, Mar 8, 2018 at 2:07 AM Ismaël Mejía <ieme...@gmail.com> >>>> wrote: >>>> >>>> >>>>>>> I confirm that the new release fixes both problems reported >>>> previously: >>>> >>>> >>>>>>> - python package name >>>> >>>>>>> - nexmark query 10 mutability issue with the direct runner. >>>> >>>> >>>>>>> One extra regression is that the the fix produced a way longer >>>> >>>>>>> execution time on the query. >>>> >>>>>>> Not sure if a blocker but worth tracking. >>>> >>>> >>>>>>> Query 10 - Batch/Bounded >>>> >>>>>>> Version Runtime(sec) Events(/sec) Results >>>> >>>>>>> 2.3.0 3.6 27609.1 1 >>>> >>>>>>> 2.4.0 30.8 3244.3 1 >>>> >>>> >>>>>>> Query 10 - Streaming/Unbounded >>>> >>>>>>> Version Runtime(sec) Events(/sec) Results >>>> >>>>>>> 2.3.0 6.3 15873.0 1 >>>> >>>>>>> 2.4.0 101.1 989.4 1 >>>> >>>> >>>>>>> On Thu, Mar 8, 2018 at 8:54 AM, Romain Manni-Bucau >>>> >>>>>>> <rmannibu...@gmail.com> wrote: >>>> >>>>>>> > -1: >>>> >>>>>>> > a) still consider waitUntilFinish broken and a big blocker >>>> >>>>>>> > b) restrictiontracker api changed and is not backward >>>> compatible >>>> >>>>>>> > ( >>>> >>>> https://github.com/apache/beam/commit/e0034314ad196d2274cef9831ed63e090bf4d4c1#diff-098d7247eb1e9d9423bfa2ae2da38a9d >>>> ) >>>> >>>>>>> > >>>> >>>>>>> > with workarounds and fixes for these two issues the other >>>> parts >>>> work (spark, >>>> >>>>>>> > flink, direct runner, java core) on my projects >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > Romain Manni-Bucau >>>> >>>>>>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>> >>>>>>> > >>>> >>>>>>> > 2018-03-08 6:26 GMT+01:00 Robert Bradshaw < >>>> rober...@google.com>: >>>> >>>>>>> >> >>>> >>>>>>> >> Hi everyone, >>>> >>>>>>> >> >>>> >>>>>>> >> Please review and vote on the release candidate #2 for the >>>> version 2.4.0, >>>> >>>>>>> >> as follows: >>>> >>>>>>> >> [ ] +1, Approve the release >>>> >>>>>>> >> [ ] -1, Do not approve the release (please provide specific >>>> comments) >>>> >>>>>>> >> >>>> >>>>>>> >> The complete staging area is available for your review, which >>>> includes: >>>> >>>>>>> >> * JIRA release notes [1], >>>> >>>>>>> >> * the official Apache source release to be deployed to >>>> dist.apache.org >>>> >>>>>>> >> [2], >>>> >>>>>>> >> which is signed with the key with fingerprint BDC9 89B0 1BD2 >>>> A463 6010 >>>> >>>>>>> >> A1CA 8F15 5E09 610D 69FB [3], >>>> >>>>>>> >> * all artifacts to be deployed to the Maven Central >>>> Repository >>>> [4], >>>> >>>>>>> >> * source code tag "v2.4.0-RC2" [5], >>>> >>>>>>> >> * website pull request listing the release and publishing >>>> the API >>>> >>>>>>> >> reference >>>> >>>>>>> >> manual [6]. >>>> >>>>>>> >> * Java artifacts were built with Maven 3.2.5 and OpenJDK >>>> 1.8.0_112. >>>> >>>>>>> >> * Python artifact are deployed along with the source release >>>> to >>>> the >>>> >>>>>>> >> dist.apache.org [2]. If I am able to figure out how to >>>> build the >>>> wheels, I >>>> >>>>>>> >> will post them there as well. >>>> >>>>>>> >> >>>> >>>>>>> >> The vote will be open for at least 72 hours. It is adopted by >>>> majority >>>> >>>>>>> >> approval, with at least 3 PMC affirmative votes. >>>> >>>>>>> >> >>>> >>>>>>> >> Thanks, >>>> >>>>>>> >> - Robert >>>> >>>>>>> >> >>>> >>>>>>> >> [1] >>>> >>>>>>> >> >>>> >>>>>>> >> >>>> >>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342682&projectId=12319527 >>>> >>>>>>> >> [2] https://dist.apache.org/repos/dist/dev/beam/2.4.0/ >>>> >>>>>>> >> [3] https://dist.apache.org/repos/dist/dev/beam/KEYS >>>> >>>>>>> >> [4] >>>> https://repository.apache.org/content/repositories/orgapachebeam-1030/ >>>> >>>>>>> >> [5] https://github.com/apache/beam/tree/v2.4.0-RC2 >>>> >>>>>>> >> [6] https://github.com/apache/beam-site/pull/398 >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>> >>> >> >
