I don't believe so, please create one (we can dedup if we happen to find another issue).
Even better if you can contribute to fix this :) Thanks, Cham On Tue, Nov 26, 2019 at 7:07 PM Chuck Yang <chuck.y...@getcruise.com> wrote: > Has anyone looked into implementing this for the Python SDK? It would > be nice to have it if only for the ability to write float values with > NaN and infinity values. I didn't see anything in Jira, happy to > create a ticket, but wanted to ask around first. > > On Thu, Oct 17, 2019 at 12:53 PM Reuven Lax <re...@google.com> wrote: > > > > I'll take a look as well. Thanks for doing this! > > > > On Fri, Oct 4, 2019 at 9:16 PM Pablo Estrada <pabl...@google.com> wrote: > >> > >> Thanks Steve! > >> I'll take a look next week. Sorry about the delay so far. > >> Best > >> -P. > >> > >> On Fri, Sep 27, 2019 at 10:37 AM Steve Niemitz <sniem...@apache.org> > wrote: > >>> > >>> I put up a semi-WIP pull request > https://github.com/apache/beam/pull/9665 for this. The initial results > look good. I'll spend some time soon adding unit tests and documentation, > but I'd appreciate it if someone could take a first pass over it. > >>> > >>> On Wed, Sep 18, 2019 at 6:14 PM Pablo Estrada <pabl...@google.com> > wrote: > >>>> > >>>> Thanks for offering to work on this! It would be awesome to have it. > I can say that we don't have that for Python ATM. > >>>> > >>>> On Mon, Sep 16, 2019 at 10:56 AM Steve Niemitz <sniem...@apache.org> > wrote: > >>>>> > >>>>> Our experience has actually been that avro is more efficient than > even parquet, but that might also be skewed from our datasets. > >>>>> > >>>>> I might try to take a crack at this, I found > https://issues.apache.org/jira/browse/BEAM-2879 tracking it (which > coincidentally references my thread from a couple years ago on the read > side of this :) ). > >>>>> > >>>>> On Mon, Sep 16, 2019 at 1:38 PM Reuven Lax <re...@google.com> wrote: > >>>>>> > >>>>>> It's been talked about, but nobody's done anything. There as some > difficulties related to type conversion (json and avro don't support the > same types), but if those are overcome then an avro version would be much > more efficient. I believe Parquet files would be even more efficient if you > wanted to go that path, but there might be more code to write (as we > already have some code in the codebase to convert between TableRows and > Avro). > >>>>>> > >>>>>> Reuven > >>>>>> > >>>>>> On Mon, Sep 16, 2019 at 10:33 AM Steve Niemitz <sniem...@apache.org> > wrote: > >>>>>>> > >>>>>>> Has anyone investigated using avro rather than json to load data > into BigQuery using BigQueryIO (+ FILE_LOADS)? > >>>>>>> > >>>>>>> I'd be interested in enhancing it to support this, but I'm curious > if there's any prior work here. > > -- > > > *Confidentiality Note:* We care about protecting our proprietary > information, confidential material, and trade secrets. This message may > contain some or all of those things. Cruise will suffer material harm if > anyone other than the intended recipient disseminates or takes any action > based on this message. If you have received this message (including any > attachments) in error, please delete it immediately and notify the sender > promptly. >
