Le jeudi 08 mars 2018 à 20:01 +0000, Reuven Lax a écrit :
> Does Nexmark use SerializableCoder?
Actually SerializableCoder is registered in Nexmark but the default and the
current configuration are set to use "by
hand serialization". See:
public static void setupPipeline(CoderStrategy coderStrategy, Pipeline p) {
CoderRegistry registry = p.getCoderRegistry(); switch (coderStrategy) {
case HAND:
registry.registerCoderForClass(Auction.class, Auction.CODER);
registry.registerCoderForClass(AuctionBid.class, AuctionBid.CODER);
registry.registerCoderForClass(AuctionCount.class, AuctionCount.CODER);
registry.registerCoderForClass(AuctionPrice.class, AuctionPrice.CODER);
registry.registerCoderForClass(Bid.class, Bid.CODER);
registry.registerCoderForClass(CategoryPrice.class, CategoryPrice.CODER);
registry.registerCoderForClass(Event.class, Event.CODER);
registry.registerCoderForClass(IdNameReserve.class, IdNameReserve.CODER);
registry.registerCoderForClass(NameCityStateId.class, NameCityStateId.CODER);
registry.registerCoderForClass(Person.class, Person.CODER);
registry.registerCoderForClass(SellerPrice.class, SellerPrice.CODER);
registry.registerCoderForClass(Done.class, Done.CODER);
registry.registerCoderForClass(BidsPerSession.class, BidsPerSession.CODER);
break; case AVRO:
registry.registerCoderProvider(AvroCoder.getCoderProvider()); break;
case JAVA:
registry.registerCoderProvider(SerializableCoder.getCoderProvider());
break; }
}
Etienne
>
> On Thu, Mar 8, 2018 at 10:42 AM Robert Bradshaw <[email protected]> wrote:>
> > I put the validation checklist spreadsheet is up at
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475>
> > > > Regarding the direct runner regression on query 10, this is
> understandable given how mutation detection has been changed for serializable
> coders (and should be tracked, probably fixed by avoiding SerializableCoder).
> It should not affect other runners. Could you file a bug?
> > > > Regarding > > waitUntilFinish, this is a bug but not a blocker--it's
> > > > been this way since teardown was introduced. There are many
> > > > nice-to-haves that one could merge from master to the release branch,
> > > > but we've seen where that trend leads.
> > Regarding the backwards incompatible changes in restriction tracker, this
> > is (as I understand it) a change to the experimental SDF API. Eugene, do
> > you want to comment on this?
> >
> > On Thu, Mar 8, 2018 at 2:07 AM Ismaël Mejía <[email protected]> wrote:> > >
> > I confirm that the new release fixes both problems reported previously:
> > >
> > >
> > > - python package name
> > >
> > > - nexmark query 10 mutability issue with the direct runner.
> > >
> > >
> > > One extra regression is that the the fix produced a way longer
> > >
> > > execution time on the query.
> > >
> > > Not sure if a blocker but worth tracking.
> > >
> > >
> > > Query 10 - Batch/Bounded
> > >
> > > Version Runtime(sec) Events(/sec) Results
> > >
> > > 2.3.0 3.6 27609.1 1
> > >
> > > 2.4.0 30.8 3244.3 1
> > >
> > >
> > > Query 10 - Streaming/Unbounded
> > >
> > > Version Runtime(sec) Events(/sec) Results
> > >
> > > 2.3.0 6.3 15873.0 1
> > >
> > > 2.4.0 101.1 989.4 1
> > >
> > >
> > > On Thu, Mar 8, 2018 at 8:54 AM, Romain Manni-Bucau
> > >
> > > <[email protected]> wrote:
> > >
> > > > -1:
> > >
> > > > a) still consider waitUntilFinish broken and a big blocker
> > >
> > > > b) restrictiontracker api changed and is not backward compatible
> > >
> > > > (https://github.com/apache/beam/commit/e0034314ad196d2274cef9831ed63e090bf4d4c1#diff-098d7247eb1e9d9423bfa2ae2da38a9d)
> > >
> > > >
> > >
> > > > with workarounds and fixes for these two issues the other parts work
> > > > (spark,
> > >
> > > > flink, direct runner, java core) on my projects
> > >
> > > >
> > >
> > > >
> > >
> > > >
> > >
> > > > Romain Manni-Bucau
> > >
> > > > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book
> > >
> > > >
> > >
> > > > 2018-03-08 6:26 GMT+01:00 Robert Bradshaw <[email protected]>:
> > >
> > > >>
> > >
> > > >> Hi everyone,
> > >
> > > >>
> > >
> > > >> Please review and vote on the release candidate #2 for the version
> > > >> 2.4.0,
> > >
> > > >> as follows:
> > >
> > > >> [ ] +1, Approve the release
> > >
> > > >> [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > > >>
> > >
> > > >> The complete staging area is available for your review, which includes:
> > >
> > > >> * JIRA release notes [1],
> > >
> > > >> * the official Apache source release to be deployed to dist.apache.org
> > >
> > > >> [2],
> > >
> > > >> which is signed with the key with fingerprint BDC9 89B0 1BD2 A463 6010
> > >
> > > >> A1CA 8F15 5E09 610D 69FB [3],
> > >
> > > >> * all artifacts to be deployed to the Maven Central Repository [4],
> > >
> > > >> * source code tag "v2.4.0-RC2" [5],
> > >
> > > >> * website pull request listing the release and publishing the API
> > >
> > > >> reference
> > >
> > > >> manual [6].
> > >
> > > >> * Java artifacts were built with Maven 3.2.5 and OpenJDK 1.8.0_112.
> > >
> > > >> * Python artifact are deployed along with the source release to the
> > >
> > > >> dist.apache.org [2]. If I am able to figure out how to build the
> > > >> wheels, I
> > >
> > > >> will post them there as well.
> > >
> > > >>
> > >
> > > >> The vote will be open for at least 72 hours. It is adopted by majority
> > >
> > > >> approval, with at least 3 PMC affirmative votes.
> > >
> > > >>
> > >
> > > >> Thanks,
> > >
> > > >> - Robert
> > >
> > > >>
> > >
> > > >> [1]
> > >
> > > >>
> > >
> > > >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342682&projectId=12319527
> > >
> > > >> [2] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
> > >
> > > >> [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
> > >
> > > >> [4]
> > > >> https://repository.apache.org/content/repositories/orgapachebeam-1030/
> > >
> > > >> [5] https://github.com/apache/beam/tree/v2.4.0-RC2
> > >
> > > >> [6] https://github.com/apache/beam-site/pull/398
> > >
> > > >
> > >
> > > >
> > >
> >
>
