Hi Robert,
Sorry for the late reply on this. I think you should be able to do this by
registering it in your pipeline's SchemaRegistry manually, like so:
Pipeline p;
p.getSchemaRegistry().registerSchemaProvider(Fx.class,
ProtoMessageSchema.class);
Of course this isn't quite as nice as just adding the DefualtSchema
annotation to a class you control. Maybe we should consider some global
config that would always use schemas for proto-generated classes.
Brian
On Sun, Jul 12, 2020 at 9:56 AM Kaymak, Tobias <tobias.kay...@ricardo.ch>
wrote:
> This sounds like it is related to the problem I'm trying to solve. (In my
> case having a Java POJO containing a protobuf backed-class and trying to
> generate a Beam Schema from it.)
>
> I would be very interested to a solution to this as well :)
>
> On Tue, Jul 7, 2020 at 2:22 PM <robert.butc...@natwestmarkets.com> wrote:
>
>> Hi All,
>>
>>
>>
>> I have a BEAM pipeline where I am reading data from some parquet files
>> and converting them into a different format based on protobuf generated
>> classes.
>>
>>
>>
>> I wish to associate a schema (derived from the protobuf classes) for my
>> PCollections. What is the appropriate way to do this with
>> protobuf-generated classes?
>>
>>
>>
>> Code excerpt:
>>
>>
>>
>> PCollection<Fx> result = input.apply("FXFilePattern", FileIO.*match*
>> ().filepattern(fxDataFilePattern))
>> .apply("FXReadMatches", FileIO.*readMatches*())
>> .apply("FXReadParquetFile", ParquetIO.*readFiles*(fxAvroSchema))
>> .apply("MapFXToProto", ParDo.*of*(MapFXToProto.*of*()));
>> boolean hasSchema = result.hasSchema(); // returns false
>>
>>
>>
>> With thanks in advance.
>>
>>
>>
>> Kind regards,
>>
>>
>>
>> Rob
>>
>>
>>
>> *Robert Butcher*
>>
>> *Technical Architect | Foundry/SRS | NatWest Markets*
>>
>> WeWork, 10 Devonshire Square, London, EC2M 4AE
>>
>> Mobile +44 (0) 7414 730866 <+44%207414%20730866>
>>
>>
>>
>> This email is classified as *CONFIDENTIAL* unless otherwise stated.
>>
>>
>>
>> This communication and any attachments are confidential and intended
>> solely for the addressee. If you are not the intended recipient please
>> advise us immediately and delete it. Unless specifically stated in the
>> message or otherwise indicated, you may not duplicate, redistribute or
>> forward this message and any attachments are not intended for distribution
>> to, or use by any person or entity in any jurisdiction or country where
>> such distribution or use would be contrary to local law or regulation.
>> NatWest Markets Plc or any affiliated entity ("NatWest Markets") accepts
>> no responsibility for any changes made to this message after it was sent.
>> Unless otherwise specifically indicated, the contents of this
>> communication and its attachments are for information purposes only and
>> should not be regarded as an offer or solicitation to buy or sell a product
>> or service, confirmation of any transaction, a valuation, indicative price
>> or an official statement. Trading desks may have a position or interest
>> that is inconsistent with any views expressed in this message. In
>> evaluating the information contained in this message, you should know that
>> it could have been previously provided to other clients and/or internal
>> NatWest Markets personnel, who could have already acted on it.
>> NatWest Markets cannot provide absolute assurances that all electronic
>> communications (sent or received) are secure, error free, not corrupted,
>> incomplete or virus free and/or that they will not be lost, mis-delivered,
>> destroyed, delayed or intercepted/decrypted by others. Therefore NatWest
>> Markets disclaims all liability with regards to electronic communications
>> (and the contents therein) if they are corrupted, lost destroyed, delayed,
>> incomplete, mis-delivered, intercepted, decrypted or otherwise
>> misappropriated by others.
>> Any electronic communication that is conducted within or through NatWest
>> Markets systems will be subject to being archived, monitored and produced
>> to regulators and in litigation in accordance with NatWest Markets’ policy
>> and local laws, rules and regulations. Unless expressly prohibited by local
>> law, electronic communications may be archived in countries other than the
>> country in which you are located, and may be treated in accordance with the
>> laws and regulations of the country of each individual included in the
>> entire chain.
>> Copyright NatWest Markets Plc. All rights reserved. See
>> https://www.nwm.com/disclaimer for further risk disclosure.
>>
>
