With providers there is also an ordering issue since multiple providers could work for a given type so we want to apply them using some stable ordering.
On Wed, Aug 19, 2020 at 10:08 AM Brian Hulette <bhule...@google.com> wrote: > Ah yes, the SchemaRegistry and SchemaProvider follow the same model, but > none of the SchemaProviders are registered by default. Users can register > the proto schema provider with > registerSchemaProvider(Class<SchemaProvider>) [1]: > > p.getSchemaRegistry().registerSchemaProvider(ProtoMessageSchema.class); > > Then SchemaCoder should be used for all proto classes. > We could use ServiceLoader to register all schema providers, then users > wouldn't need to do this. I assume the reason we don't already is because > schemas are still experimental and we want it to be opt-in. > > [1] > https://beam.apache.org/releases/javadoc/2.23.0/org/apache/beam/sdk/schemas/SchemaRegistry.html#registerSchemaProvider-org.apache.beam.sdk.schemas.SchemaProvider- > <https://beam.apache.org/releases/javadoc/2.23.0/org/apache/beam/sdk/schemas/SchemaRegistry.html#registerSchemaProvider-org.apache.beam.sdk.schemas.SchemaProvider-> > > On Wed, Aug 19, 2020 at 8:44 AM Luke Cwik <lc...@google.com> wrote: > >> Brian, Coders have a provider model where the provider can be queried to >> resolve for a given type and the providers are resolved in a specific >> order. This gave the flexibility to handle situations like the one you >> described. >> >> On Wed, Aug 19, 2020 at 12:30 AM <robert.butc...@natwestmarkets.com> >> wrote: >> >>> Hi Brian, >>> >>> >>> >>> Many thanks for your mail. >>> >>> >>> >>> Yes I figured that one out in the end from the docs, but many thanks for >>> confirming. >>> >>> >>> >>> I did subsequently discover some other issues with protoBuf-derived >>> schemas (essentially they don’t seem to be properly supported by >>> BigQueryIO.Write or allow for optional fields) but I posted a separate >>> message on the dev channel covering this. >>> >>> >>> >>> Kind regards, >>> >>> >>> >>> Rob >>> >>> >>> >>> *From:* Brian Hulette [mailto:bhule...@google.com] >>> *Sent:* 18 August 2020 20:50 >>> *To:* user >>> *Subject:* Re: Registering Protobuf schema >>> >>> >>> >>> >>> ********************************************* >>> "This is an external email. Do you know who has sent it? Can you be sure >>> that any links and attachments contained within it are safe? If in any >>> doubt, use the Phishing Reporter Button in your Outlook client or forward >>> the email as an attachment to ~ I've Been Phished" >>> ********************************************* >>> >>> Hi Robert, >>> Sorry for the late reply on this. I think you should be able to do this >>> by registering it in your pipeline's SchemaRegistry manually, like so: >>> >>> >>> >>> Pipeline p; >>> >>> p.getSchemaRegistry().registerSchemaProvider(Fx.class, >>> ProtoMessageSchema.class); >>> >>> Of course this isn't quite as nice as just adding the DefualtSchema >>> annotation to a class you control. Maybe we should consider some global >>> config that would always use schemas for proto-generated classes. >>> >>> >>> Brian >>> >>> >>> >>> On Sun, Jul 12, 2020 at 9:56 AM Kaymak, Tobias <tobias.kay...@ricardo.ch> >>> wrote: >>> >>> This sounds like it is related to the problem I'm trying to solve. (In >>> my case having a Java POJO containing a protobuf backed-class and trying to >>> generate a Beam Schema from it.) >>> >>> I would be very interested to a solution to this as well :) >>> >>> >>> >>> On Tue, Jul 7, 2020 at 2:22 PM <robert.butc...@natwestmarkets.com> >>> wrote: >>> >>> Hi All, >>> >>> >>> >>> I have a BEAM pipeline where I am reading data from some parquet files >>> and converting them into a different format based on protobuf generated >>> classes. >>> >>> >>> >>> I wish to associate a schema (derived from the protobuf classes) for my >>> PCollections. What is the appropriate way to do this with >>> protobuf-generated classes? >>> >>> >>> >>> Code excerpt: >>> >>> >>> >>> PCollection<Fx> result = input.apply("FXFilePattern", FileIO.*match* >>> ().filepattern(fxDataFilePattern)) >>> .apply("FXReadMatches", FileIO.*readMatches*()) >>> .apply("FXReadParquetFile", ParquetIO.*readFiles*(fxAvroSchema)) >>> .apply("MapFXToProto", ParDo.*of*(MapFXToProto.*of*())); >>> boolean hasSchema = result.hasSchema(); // returns false >>> >>> >>> >>> With thanks in advance. >>> >>> >>> >>> Kind regards, >>> >>> >>> >>> Rob >>> >>> >>> >>> *Robert Butcher* >>> >>> *Technical Architect | Foundry/SRS | NatWest Markets* >>> >>> WeWork, 10 Devonshire Square, London, EC2M 4AE >>> >>> Mobile +44 (0) 7414 730866 <+44%207414%20730866> >>> >>> >>> >>> This email is classified as *CONFIDENTIAL* unless otherwise stated. >>> >>> >>> >>> >>> >>> This communication and any attachments are confidential and intended >>> solely for the addressee. If you are not the intended recipient please >>> advise us immediately and delete it. Unless specifically stated in the >>> message or otherwise indicated, you may not duplicate, redistribute or >>> forward this message and any attachments are not intended for distribution >>> to, or use by any person or entity in any jurisdiction or country where >>> such distribution or use would be contrary to local law or regulation. >>> NatWest Markets Plc or any affiliated entity ("NatWest Markets") accepts >>> no responsibility for any changes made to this message after it was sent. >>> >>> Unless otherwise specifically indicated, the contents of this >>> communication and its attachments are for information purposes only and >>> should not be regarded as an offer or solicitation to buy or sell a product >>> or service, confirmation of any transaction, a valuation, indicative price >>> or an official statement. Trading desks may have a position or interest >>> that is inconsistent with any views expressed in this message. In >>> evaluating the information contained in this message, you should know that >>> it could have been previously provided to other clients and/or internal >>> NatWest Markets personnel, who could have already acted on it. >>> >>> NatWest Markets cannot provide absolute assurances that all electronic >>> communications (sent or received) are secure, error free, not corrupted, >>> incomplete or virus free and/or that they will not be lost, mis-delivered, >>> destroyed, delayed or intercepted/decrypted by others. Therefore NatWest >>> Markets disclaims all liability with regards to electronic communications >>> (and the contents therein) if they are corrupted, lost destroyed, delayed, >>> incomplete, mis-delivered, intercepted, decrypted or otherwise >>> misappropriated by others. >>> >>> Any electronic communication that is conducted within or through NatWest >>> Markets systems will be subject to being archived, monitored and produced >>> to regulators and in litigation in accordance with NatWest Markets’ policy >>> and local laws, rules and regulations. Unless expressly prohibited by local >>> law, electronic communications may be archived in countries other than the >>> country in which you are located, and may be treated in accordance with the >>> laws and regulations of the country of each individual included in the >>> entire chain. >>> >>> Copyright NatWest Markets Plc. All rights reserved. See >>> https://www.nwm.com/disclaimer for further risk disclosure. >>> >>> >>> This communication and any attachments are confidential and intended >>> solely for the addressee. If you are not the intended recipient please >>> advise us immediately and delete it. Unless specifically stated in the >>> message or otherwise indicated, you may not duplicate, redistribute or >>> forward this message and any attachments are not intended for distribution >>> to, or use by any person or entity in any jurisdiction or country where >>> such distribution or use would be contrary to local law or regulation. >>> NatWest Markets Plc, NatWest Markets N.V., NatWest Markets Securities Japan >>> Limited and/or NatWest Markets Securities Inc. (collectively "NatWest >>> Markets") accepts no responsibility for any changes made to this message >>> after it was sent. >>> This communication, where prepared by the sales and trading desk or desk >>> strategists, may be marketing material, desk strategy and/or trader >>> commentary. It is not a product of the research department. This material >>> may constitute an invitation to consider entering into a derivatives >>> transaction under U.S. CFTC Regulations sections 1.71 and 23.605, where >>> applicable, but is not a binding offer to buy/sell any financial >>> instrument. The views of the author may differ from others at NatWest >>> Markets. >>> Unless otherwise specifically indicated, the contents of this >>> communication and its attachments are for information purposes only and >>> should not be regarded as an offer or solicitation to buy or sell a product >>> or service, confirmation of any transaction, a valuation, indicative price >>> or an official statement. Trading desks may have a position or interest >>> that is inconsistent with any views expressed in this message. In >>> evaluating the information contained in this message, you should know that >>> it could have been previously provided to other clients and/or internal >>> NatWest Markets personnel, who could have already acted on it. >>> NatWest Markets cannot provide absolute assurances that all electronic >>> communications (sent or received) are secure, error free, not corrupted, >>> incomplete or virus free and/or that they will not be lost, mis-delivered, >>> destroyed, delayed or intercepted/decrypted by others. Therefore NatWest >>> Markets disclaims all liability with regards to electronic communications >>> (and the contents therein) if they are corrupted, lost destroyed, delayed, >>> incomplete, mis-delivered, intercepted, decrypted or otherwise >>> misappropriated by others. >>> Any electronic communication that is conducted within or through NatWest >>> Markets systems will be subject to being archived, monitored and produced >>> to regulators and in litigation in accordance with NatWest Markets’ policy >>> and local laws, rules and regulations. Unless expressly prohibited by local >>> law, electronic communications may be archived in countries other than the >>> country in which you are located, and may be treated in accordance with the >>> laws and regulations of the country of each individual included in the >>> entire chain. >>> Copyright © NatWest Markets Plc. All rights reserved. See >>> https://www.nwm.com/disclaimer for further risk disclosure (the agency >>> arrangements referred to in the further risk disclosure between NatWest >>> Markets Plc and NatWest Markets N.V. are not applicable to branches of >>> NatWest Markets N.V.). >>> >>
