Thank you Valentyn for reporting this. I have traced the issue back to https://github.com/apache/beam/pull/4666, so I have sent out a PR to fix: https://github.com/apache/beam/pull/4846.
On Fri, Mar 9, 2018 at 2:17 PM, Valentyn Tymofieiev <[email protected]> wrote: > -1. > > Checked Python Quickstarts (Passed) and Python MobileGaming on > DirectRunner. I observe an issue in BQ sink for hourly teams score example: > > https://issues.apache.org/jira/browse/BEAM-3824 > > On Fri, Mar 9, 2018 at 10:49 AM, Lukasz Cwik <[email protected]> wrote: > >> I checked that word count quickstarts (except Dataflow) worked for RC2 to >> hopefully prevent an RC4. >> >> On Fri, Mar 9, 2018 at 10:29 AM, Robert Bradshaw <[email protected]> >> wrote: >> >>> Thanks, Alan, for pointing this out. I see this now, and it looks like I >>> need to finish building the dataflow workers so they have something to >>> point to. I will do this and release an RC3 once that's ready. >>> >>> In the meantime, it'd be great if we could validate everything else about >>> this RC such that when this on-line, dataflow-only change is out we won't >>> have any further surprises. I see Luke went through the Java Quickstart >>> examples, thanks! >>> >>> >>> On Thu, Mar 8, 2018 at 3:48 PM Lukasz Cwik <[email protected]> wrote: >>> >>> > Yes, the release guide has a segment "Update release specific >>> configurations" that has a tidbit about this. >>> >>> > On Thu, Mar 8, 2018 at 3:45 PM, Alan Myrvold <[email protected]> >>> wrote: >>> >>> >> The dataflow java worker version wasn't updated on the branch as in >>> past >>> releases ... should it be? >>> >> https://issues.apache.org/jira/browse/BEAM-3815 >>> >>> >>> >> On Thu, Mar 8, 2018 at 1:40 PM Romain Manni-Bucau < >>> [email protected]> >>> wrote: >>> >>> >>> Can still be provided as a generic one (like the an offset or key >>> based >>> one) but good enough for now, right, was just surprising to not see it >>> when >>> checking the breakage. >>> >>> >>> Le 8 mars 2018 22:05, "Eugene Kirpichov" <[email protected]> a >>> écrit >>> : >>> >>> >>> All SDF-related method annotations in DoFn are marked @Experimental. >>> I >>> guess that should apply to RestrictionTracker too, but I wouldn't be too >>> worried about that, since it only makes sense to use in the context of >>> those methods. >>> >>> >>> On Thu, Mar 8, 2018 at 12:36 PM Romain Manni-Bucau < >>> [email protected]> wrote: >>> >>> >>>> Hmm, does sdf api misses some @Experimental then? >>> >>> >>>> To clarify: for waitUntilFinish I'm fine with the 2.4 as this but >>> cant >>> +1 or +0 since none of my tests pass reliably in current state without a >>> retry strategy making the call useless. >>> >>> >>>> Le 8 mars 2018 21:02, "Reuven Lax" <[email protected]> a écrit : >>> >>> >>>>> Does Nexmark use SerializableCoder? >>> >>> >>> >>>>> On Thu, Mar 8, 2018 at 10:42 AM Robert Bradshaw < >>> [email protected]> >>> wrote: >>> >>> >>>>>> I put the validation checklist spreadsheet is up at >>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkS >>> ZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475 >>> >>> >>>>>> Regarding the direct runner regression on query 10, this is >>> understandable given how mutation detection has been changed for >>> serializable coders (and should be tracked, probably fixed by avoiding >>> SerializableCoder). It should not affect other runners. Could you file a >>> bug? >>> >>> >>>>>> Regarding waitUntilFinish, this is a bug but not a blocker--it's >>> been this way since teardown was introduced. There are many nice-to-haves >>> that one could merge from master to the release branch, but we've seen >>> where that trend leads. >>> >>> >>>>>> Regarding the backwards incompatible changes in restriction >>> tracker, >>> this is (as I understand it) a change to the experimental SDF API. >>> Eugene, >>> do you want to comment on this? >>> >>> >>> >>> >>>>>> On Thu, Mar 8, 2018 at 2:07 AM Ismaël Mejía <[email protected]> >>> wrote: >>> >>> >>>>>>> I confirm that the new release fixes both problems reported >>> previously: >>> >>> >>>>>>> - python package name >>> >>>>>>> - nexmark query 10 mutability issue with the direct runner. >>> >>> >>>>>>> One extra regression is that the the fix produced a way longer >>> >>>>>>> execution time on the query. >>> >>>>>>> Not sure if a blocker but worth tracking. >>> >>> >>>>>>> Query 10 - Batch/Bounded >>> >>>>>>> Version Runtime(sec) Events(/sec) Results >>> >>>>>>> 2.3.0 3.6 27609.1 1 >>> >>>>>>> 2.4.0 30.8 3244.3 1 >>> >>> >>>>>>> Query 10 - Streaming/Unbounded >>> >>>>>>> Version Runtime(sec) Events(/sec) Results >>> >>>>>>> 2.3.0 6.3 15873.0 1 >>> >>>>>>> 2.4.0 101.1 989.4 1 >>> >>> >>>>>>> On Thu, Mar 8, 2018 at 8:54 AM, Romain Manni-Bucau >>> >>>>>>> <[email protected]> wrote: >>> >>>>>>> > -1: >>> >>>>>>> > a) still consider waitUntilFinish broken and a big blocker >>> >>>>>>> > b) restrictiontracker api changed and is not backward >>> compatible >>> >>>>>>> > ( >>> https://github.com/apache/beam/commit/e0034314ad196d2274cef9 >>> 831ed63e090bf4d4c1#diff-098d7247eb1e9d9423bfa2ae2da38a9d >>> ) >>> >>>>>>> > >>> >>>>>>> > with workarounds and fixes for these two issues the other parts >>> work (spark, >>> >>>>>>> > flink, direct runner, java core) on my projects >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > Romain Manni-Bucau >>> >>>>>>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>> >>>>>>> > >>> >>>>>>> > 2018-03-08 6:26 GMT+01:00 Robert Bradshaw <[email protected] >>> >: >>> >>>>>>> >> >>> >>>>>>> >> Hi everyone, >>> >>>>>>> >> >>> >>>>>>> >> Please review and vote on the release candidate #2 for the >>> version 2.4.0, >>> >>>>>>> >> as follows: >>> >>>>>>> >> [ ] +1, Approve the release >>> >>>>>>> >> [ ] -1, Do not approve the release (please provide specific >>> comments) >>> >>>>>>> >> >>> >>>>>>> >> The complete staging area is available for your review, which >>> includes: >>> >>>>>>> >> * JIRA release notes [1], >>> >>>>>>> >> * the official Apache source release to be deployed to >>> dist.apache.org >>> >>>>>>> >> [2], >>> >>>>>>> >> which is signed with the key with fingerprint BDC9 89B0 1BD2 >>> A463 6010 >>> >>>>>>> >> A1CA 8F15 5E09 610D 69FB [3], >>> >>>>>>> >> * all artifacts to be deployed to the Maven Central Repository >>> [4], >>> >>>>>>> >> * source code tag "v2.4.0-RC2" [5], >>> >>>>>>> >> * website pull request listing the release and publishing the >>> API >>> >>>>>>> >> reference >>> >>>>>>> >> manual [6]. >>> >>>>>>> >> * Java artifacts were built with Maven 3.2.5 and OpenJDK >>> 1.8.0_112. >>> >>>>>>> >> * Python artifact are deployed along with the source release >>> to >>> the >>> >>>>>>> >> dist.apache.org [2]. If I am able to figure out how to build >>> the >>> wheels, I >>> >>>>>>> >> will post them there as well. >>> >>>>>>> >> >>> >>>>>>> >> The vote will be open for at least 72 hours. It is adopted by >>> majority >>> >>>>>>> >> approval, with at least 3 PMC affirmative votes. >>> >>>>>>> >> >>> >>>>>>> >> Thanks, >>> >>>>>>> >> - Robert >>> >>>>>>> >> >>> >>>>>>> >> [1] >>> >>>>>>> >> >>> >>>>>>> >> >>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?versi >>> on=12342682&projectId=12319527 >>> >>>>>>> >> [2] https://dist.apache.org/repos/dist/dev/beam/2.4.0/ >>> >>>>>>> >> [3] https://dist.apache.org/repos/dist/dev/beam/KEYS >>> >>>>>>> >> [4] >>> https://repository.apache.org/content/repositories/orgapachebeam-1030/ >>> >>>>>>> >> [5] https://github.com/apache/beam/tree/v2.4.0-RC2 >>> >>>>>>> >> [6] https://github.com/apache/beam-site/pull/398 >>> >>>>>>> > >>> >>>>>>> > >>> >> >> >
