Re: Dynamically generate BigQuery schema based on input?

Here are the JIRAs. BigQuery custom sink: https://issues.apache.org/jira/browse/BEAM-2801 BigQuery custom source: https://issues.apache.org/jira/browse/BEAM-1440 Volunteers are welcome :) Thanks, Cham On Wed, Aug 23, 2017 at 7:21 PM Chamikara Jayalath wrote: > I don't

Re: Dynamically generate BigQuery schema based on input?

I don't think we have JIRAs. I'll go ahead and create them. Thanks, Cham On Wed, Aug 23, 2017 at 6:33 PM Reuven Lax wrote: > Cham, > > Do we have JIRAs tracking these features? > > Reuven > > On Wed, Aug 23, 2017 at 6:26 PM, Asha Rostamianfar < >

Re: Dynamically generate BigQuery schema based on input?

It's for a batch job. I'll think about possible workarounds then (perhaps I can have a prestage process that reads the necessary info from the files and passes as an argument). Thanks everyone! On Wed, Aug 23, 2017 at 9:17 PM, Chamikara Jayalath wrote: > Ah, actually for

Jenkins build is back to normal : beam_SeedJob #398

See

Re: Dynamically generate BigQuery schema based on input?

Is this for a batch pipeline or a streaming pipeline? I don't believe that Python yet has good support for BigQuery in batch jobs at all. On Wed, Aug 23, 2017 at 6:10 PM, Asha Rostamianfar < arost...@google.com.invalid> wrote: > ah, ok..is there an existing JIRA entry with some approximate ETA?

Re: Dynamically generate BigQuery schema based on input?

We have to develop a BigQuery BoundedSource for this first. So I expect support for Python SDK to be at least few months away. Thanks, Cham On Wed, Aug 23, 2017 at 6:10 PM Asha Rostamianfar wrote: > ah, ok..is there an existing JIRA entry with some approximate ETA?

Re: Dynamically generate BigQuery schema based on input?

Not yet, only in Java so far. On Wed, Aug 23, 2017 at 5:53 PM, Asha Rostamianfar < arost...@google.com.invalid> wrote: > Thanks, Eugene! Is this also supported in the python SDK? > > On Wed, Aug 23, 2017 at 8:39 PM, Eugene Kirpichov < > kirpic...@google.com.invalid> wrote: > > > Yes, this is

Re: Dynamically generate BigQuery schema based on input?

Thanks, Eugene! Is this also supported in the python SDK? On Wed, Aug 23, 2017 at 8:39 PM, Eugene Kirpichov < kirpic...@google.com.invalid> wrote: > Yes, this is possible using the BigQueryIO.write().to(DynamicDestinations) > API. It allows you to write different values to different tables with

Re: Dynamically generate BigQuery schema based on input?

Yes, this is possible using the BigQueryIO.write().to(DynamicDestinations) API. It allows you to write different values to different tables with different schemas. On Wed, Aug 23, 2017, 2:51 PM Asha Rostamianfar wrote: > Hi, > > I'm wondering whether it's possible to

Dynamically generate BigQuery schema based on input?

Hi, I'm wondering whether it's possible to dynamically generate a BigQuery schema based on input. For instance, the fields would be specified in one or more input files that are read and processed as part of the pipeline. Thanks, Asha

Re: [RESULT][VOTE] Release 2.1.0, release candidate #3

On Tue, Aug 22, 2017 at 5:12 PM, Ahmet Altay wrote: > I believe this release is complete now. Thank you JB for pushing this > release, and everyone else who contributed to it. > > On Tue, Aug 22, 2017 at 4:26 PM, Ahmet Altay wrote: > >> Remaining items for

Re: [DISCUSS] Capability Matrix revamp

I would like to have an API compatibility testing. AFAIK there's still gap to achieve our goal (one job for any runner), that means developers should notice the limitation when writing the job. For example PCollectionView is not well supported in FlinkRunner(not quite sure the current status as my

Re: [Proposal] Progress Reporting in Fn API

Hi Robert, Thanks for the amazing write up!, it definitely helps. The design is moving in a direction described in your doc, if not entirely there yet. Regards, Vikas On 22 August 2017 at 11:49, Robert Bradshaw wrote: > I put together

How to decrease or merge bundles (Spark Coalesce like operation) with Apache Beam

Hi Team, We read about 5k files and executed the pipeline using spark runner. Each file starts with a new pipeline, followed by various validations and transformations . After that all pipelines are aggregated using PCollectionList followed by flatten operation to create a single pCollection.

Re: Jenkins build is still unstable: beam_Release_NightlySnapshot #512

Just so these nightly build failures do not go by without comment: they are both failures in BigtableWriteIT.testE2EBigtableWrite that look like https://issues.apache.org/jira/browse/BEAM-2545 On Wed, Aug 23, 2017 at 1:25 AM, Apache Jenkins Server < jenk...@builds.apache.org> wrote: > See

Jenkins build is still unstable: beam_Release_NightlySnapshot #512

See