Here's https://issues.apache.org/jira/browse/BEAM-7064 on this topic.
Kenn On Thu, Aug 6, 2020 at 2:27 PM Chamikara Jayalath <[email protected]> wrote: > Created https://issues.apache.org/jira/browse/BEAM-10655 > Please add any additional information there. > > If you hope to provide a fix that's welcome too. Also, if you need to get > the JIRA assigned to you, you can request JIRA access through the Beam dev > list (you don't need to be a committer for this). > > Thanks, > Cham > > > On Wed, Aug 5, 2020 at 6:21 AM <[email protected]> wrote: > >> Hi Cham, >> >> >> >> I’m not registered as a BEAM committer so I don’t believe I have access >> rights to raise JIRAs. Would you be kind enough to file a bug report on my >> behalf? >> >> >> >> I would suggest creating a simple end-to-end test that takes a >> protobuf-generated domain object and writes to BigQuery making use of >> useBeamSchema (I couldn’t find such an example in the test suite). >> Briefly the issues I have identified are: >> >> 1. As reported below, schema conversion does not recognise the >> protobuf-derived special types (e.g. NanosInstant). This results in an NPE >> >> 2. The schema conversion code does not recognise fields in camel >> case (it works if fields are lower case) >> >> 3. The schema presented to BigQueryIO presents all fields as >> required (despite the fact that under protobuf3 all fields are optional) >> >> >> >> Point 3 is more likely an issue with the protobuf to BEAM schema >> generation, however I present it here as it would be useful to capture as >> part of an end-to-end test suite. >> >> >> >> Kind regards, >> >> >> >> Rob >> >> *From:* Chamikara Jayalath [mailto:[email protected]] >> *Sent:* 15 July 2020 16:36 >> *To:* dev >> *Subject:* Re: NanosInstant not being recognised by BigQueryIO.Write >> >> >> >> >> ********************************************* >> "This is an external email. Do you know who has sent it? Can you be sure >> that any links and attachments contained within it are safe? If in any >> doubt, use the Phishing Reporter Button in your Outlook client or forward >> the email as an attachment to ~ I've Been Phished" >> ********************************************* >> >> >> >> >> >> On Wed, Jul 8, 2020 at 3:30 AM <[email protected]> wrote: >> >> Hi All, >> >> >> >> I am posting this to the dev (as opposed to user channel) as I believe it >> will be of interest to the those working on either Schemas or BigQuery >> >> >> >> I have a pipeline based on BEAM 2.22 that is ingesting data into >> BigQuery. Internally I am using protobuf for my domain model and the >> associated schema support. >> >> >> >> My intention is to make use of the useBeamSchema() method to both >> auto-generate the BigQuery table schema and to provide row conversion on >> write. (The idea is to have true schema-first development very much in >> keeping with Alex’s original ProtoBEAM concept). >> >> >> >> The issue I’ve hit is around treatment of google.protobuf.Timestamp >> fields. The schema conversion seems to map these to the correct logical >> type: org.apache.beam.sdk.schemas.logicaltypes.NanosInstant, however >> this isn’t recognised by BigQueryIO.Write. Specifically the >> BigQueryUtils.toTableSchema() method throws a NullPointerException. >> This seems to be due to the fact that there is no entry for NanosInstant >> in the BEAM_TO_BIG_QUERY_LOGICAL_MAPPING map. >> >> >> >> This does sound like a bug since Beam Schema to BigQuery type conversion >> [1] indeed does not consider >> org.apache.beam.sdk.schemas.logicaltypes.NanosInstant. Will you be able to >> file a JIRA with preferably a test to reproduce this ? >> >> >> https://github.com/apache/beam/blob/6c313eb84af6229f0a8a7a0b5890f18c5a8685e8/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java#L202 >> >> >> >> Thanks, >> >> Cham >> >> >> >> >> >> Is this a known issue? Is there a workaround? >> >> >> >> I appreciate that google.protobuf.Timestamp supports nanosecond-level >> precision so cannot be converted directly to the BEAM schema type of >> DATETIME without loss of precision. However, I believe use cases for >> nanosecond precision are rare. Would it not be better to convert directly >> to DATETIME according to the *principle of least confusion*? >> >> >> >> Are there any plans to extend the range of types both within protobuf and >> the BEAM schema to match the richer type set within BigQuery (DATE, >> DATETIME, TIMESTAMP)? I would expect the combination of >> protobuf/BEAM/BigQuery to be a common one (especially within GCP) and it >> would be nice as a developer to have a greater range of options. >> >> >> >> Kind regards, >> >> >> >> Rob >> >> >> >> *Robert Butcher* >> >> *Technical Architect | Foundry/SRS | NatWest Markets* >> >> WeWork, 10 Devonshire Square, London, EC2M 4AE >> >> Mobile +44 (0) 7414 730866 <+44%207414%20730866> >> >> >> >> This email is classified as *CONFIDENTIAL* unless otherwise stated. >> >> >> >> >> >> This communication and any attachments are confidential and intended >> solely for the addressee. If you are not the intended recipient please >> advise us immediately and delete it. Unless specifically stated in the >> message or otherwise indicated, you may not duplicate, redistribute or >> forward this message and any attachments are not intended for distribution >> to, or use by any person or entity in any jurisdiction or country where >> such distribution or use would be contrary to local law or regulation. >> NatWest Markets Plc or any affiliated entity ("NatWest Markets") accepts >> no responsibility for any changes made to this message after it was sent. >> >> Unless otherwise specifically indicated, the contents of this >> communication and its attachments are for information purposes only and >> should not be regarded as an offer or solicitation to buy or sell a product >> or service, confirmation of any transaction, a valuation, indicative price >> or an official statement. Trading desks may have a position or interest >> that is inconsistent with any views expressed in this message. In >> evaluating the information contained in this message, you should know that >> it could have been previously provided to other clients and/or internal >> NatWest Markets personnel, who could have already acted on it. >> >> NatWest Markets cannot provide absolute assurances that all electronic >> communications (sent or received) are secure, error free, not corrupted, >> incomplete or virus free and/or that they will not be lost, mis-delivered, >> destroyed, delayed or intercepted/decrypted by others. Therefore NatWest >> Markets disclaims all liability with regards to electronic communications >> (and the contents therein) if they are corrupted, lost destroyed, delayed, >> incomplete, mis-delivered, intercepted, decrypted or otherwise >> misappropriated by others. >> >> Any electronic communication that is conducted within or through NatWest >> Markets systems will be subject to being archived, monitored and produced >> to regulators and in litigation in accordance with NatWest Markets’ policy >> and local laws, rules and regulations. Unless expressly prohibited by local >> law, electronic communications may be archived in countries other than the >> country in which you are located, and may be treated in accordance with the >> laws and regulations of the country of each individual included in the >> entire chain. >> >> Copyright NatWest Markets Plc. All rights reserved. See >> https://www.nwm.com/disclaimer for further risk disclosure. >> >> >> This communication and any attachments are confidential and intended >> solely for the addressee. If you are not the intended recipient please >> advise us immediately and delete it. Unless specifically stated in the >> message or otherwise indicated, you may not duplicate, redistribute or >> forward this message and any attachments are not intended for distribution >> to, or use by any person or entity in any jurisdiction or country where >> such distribution or use would be contrary to local law or regulation. >> NatWest Markets Plc, NatWest Markets N.V., NatWest Markets Securities Japan >> Limited and/or NatWest Markets Securities Inc. (collectively "NatWest >> Markets") accepts no responsibility for any changes made to this message >> after it was sent. >> This communication, where prepared by the sales and trading desk or desk >> strategists, may be marketing material, desk strategy and/or trader >> commentary. It is not a product of the research department. This material >> may constitute an invitation to consider entering into a derivatives >> transaction under U.S. CFTC Regulations sections 1.71 and 23.605, where >> applicable, but is not a binding offer to buy/sell any financial >> instrument. The views of the author may differ from others at NatWest >> Markets. >> Unless otherwise specifically indicated, the contents of this >> communication and its attachments are for information purposes only and >> should not be regarded as an offer or solicitation to buy or sell a product >> or service, confirmation of any transaction, a valuation, indicative price >> or an official statement. Trading desks may have a position or interest >> that is inconsistent with any views expressed in this message. In >> evaluating the information contained in this message, you should know that >> it could have been previously provided to other clients and/or internal >> NatWest Markets personnel, who could have already acted on it. >> NatWest Markets cannot provide absolute assurances that all electronic >> communications (sent or received) are secure, error free, not corrupted, >> incomplete or virus free and/or that they will not be lost, mis-delivered, >> destroyed, delayed or intercepted/decrypted by others. Therefore NatWest >> Markets disclaims all liability with regards to electronic communications >> (and the contents therein) if they are corrupted, lost destroyed, delayed, >> incomplete, mis-delivered, intercepted, decrypted or otherwise >> misappropriated by others. >> Any electronic communication that is conducted within or through NatWest >> Markets systems will be subject to being archived, monitored and produced >> to regulators and in litigation in accordance with NatWest Markets’ policy >> and local laws, rules and regulations. Unless expressly prohibited by local >> law, electronic communications may be archived in countries other than the >> country in which you are located, and may be treated in accordance with the >> laws and regulations of the country of each individual included in the >> entire chain. >> Copyright © NatWest Markets Plc. All rights reserved. See >> https://www.nwm.com/disclaimer for further risk disclosure (the agency >> arrangements referred to in the further risk disclosure between NatWest >> Markets Plc and NatWest Markets N.V. are not applicable to branches of >> NatWest Markets N.V.). >> >
