Just opened this PR: https://github.com/apache/beam/pull/6055 to get
feedback ASAP. Basically what it does is return the job status in a
PCollection of BigQueryWriteResult objects
On Fri, Jul 20, 2018 at 11:57 PM Reuven Lax wrote:
> There already is a org.apache.beam.sdk.io.gcp.bigquery.WriteRes
Hmm, I think this approach has some complications:
- Using JobStatus makes it tied to using BigQuery batch load jobs, but the
return type ought to be the same regardless of which method of writing is
used (including potential future BigQuery APIs - they are evolving), or how
many BigQuery load jobs
All good so far. I've been a bit side tracked but more or less I have the
idea of using the JobStatus as part of the collection so that not only the
completion is signaled, but also the result (success/failure) can be
accessed, how does it sound?
Regards
On Tue, Jul 17, 2018 at 3:07 AM Eugene Kir
Hi Carlos,
Any updates / roadblocks you hit?
On Tue, Jul 3, 2018 at 7:13 AM Eugene Kirpichov
wrote:
> Awesome!! Thanks for the heads up, very exciting, this is going to make a
> lot of people happy :)
>
> On Tue, Jul 3, 2018, 3:40 AM Carlos Alonso wrote:
>
>> + d...@beam.apache.org
>>
>> Just
Awesome!! Thanks for the heads up, very exciting, this is going to make a
lot of people happy :)
On Tue, Jul 3, 2018, 3:40 AM Carlos Alonso wrote:
> + d...@beam.apache.org
>
> Just a quick email to let you know that I'm starting developing this.
>
> On Fri, Apr 20, 2018 at 10:30 PM Eugene Kirpic
+ d...@beam.apache.org
Just a quick email to let you know that I'm starting developing this.
On Fri, Apr 20, 2018 at 10:30 PM Eugene Kirpichov
wrote:
> Hi Carlos,
>
> Thank you for expressing interest in taking this on! Let me give you a few
> pointers to start, and I'll be happy to help everyw
Hi Carlos,
Thank you for expressing interest in taking this on! Let me give you a few
pointers to start, and I'll be happy to help everywhere along the way.
Basically we want BigQueryIO.write() to return something (e.g. a
PCollection) that can be used as input to Wait.on().
Currently it returns a
Hey Eugene!!
I’d gladly take a stab on it although I’m not sure how much available time
I might have to put into but... yeah, let’s try it.
Where should I begin? Is there a Jira issue or shall I file one?
Thanks!
On Thu, 12 Apr 2018 at 00:41, Eugene Kirpichov wrote:
> Hi,
>
> Yes, you're both
Hi,
Yes, you're both right - BigQueryIO.write() is currently not implemented in
a way that it can be used with Wait.on(). It would certainly be a welcome
contribution to change this - many people expressed interest in
specifically waiting for BigQuery writes. Is any of you interested in
helping ou
Hi Simon, I think your explanation was very accurate, at least to my
understanding. I'd also be interested in getting batch load result's
feedback on the pipeline... hopefully someone may suggest something,
otherwise we could propose submitting a Jira, or even better, a PR!! :)
Thanks!
On Thu, Ap
Hi All,
I need to write some data to BigQuery (batch-mode) and then send a Pubsub
message to trigger further processing.
I found this thread titled "Callbacks/other functions run after a PDone/output
transform" on the user-list which was very relevant:
https://lists.apache.org/thread.html/dd
11 matches
Mail list logo
