How would you know how many Voids to wait for downstream? On Sat, Sep 9, 2017 at 10:46 AM, Eugene Kirpichov <[email protected]> wrote:
> Hi Steve, > Unfortunately for BigQuery it's more complicated than that. Rows aren't > written to BigQuery one by one (unless you're using streaming inserts, > which are way more expensive and are usually used only in streaming > pipelines) - they are written to files, and then a BigQuery import job, or > several import jobs if there are too many files, picks them up. We can > declare writing complete when all of the BigQuery import jobs have > successfully completed. > However, the method of writing is an implementation detail of BigQuery, so > we need to create an API that works regardless of the method (import jobs > vs. streaming inserts). > Another complication is triggering - windows can fire multiple times. This > rules out any approaches that sequence using side inputs, because side > inputs don't have triggering. > > I think a common approach could be to return a PCollection<Void>, > containing a Void in every window and pane that has been successfully > written. This could be implemented in both modes and could be a general > design patterns for this sort of thing. It just isn't easy to implement, so > I didn't have time to take it on. It also could turn out to have other > complications we haven't thought of yet. > > That said, if somebody tried to implement this for some connectors (not > necessarily BigQuery) and pioneered the approach, it would be a great > contribution. > > On Sat, Sep 9, 2017 at 9:41 AM Steve Niemitz <[email protected]> wrote: > >> I wonder if it makes sense to start simple and go from there. For >> example, >> I enhanced BigtableIO.Write to output the number of rows written >> in finishBundle(), simply into the global window with the current >> timestamp. This was more than enough to unblock me, but doesn't support >> more complicated scenarios with windowing. >> >> However, as I said it was more than enough to solve the general batch use >> case, and I imagine could be enhanced to support windowing by keeping >> track >> of which windows were written per bundle. (can there even ever be more >> than >> one window per bundle?) >> >> On Fri, Sep 8, 2017 at 2:32 PM, Eugene Kirpichov < >> [email protected]> wrote: >> >> > Hi, >> > I was going to implement this, but discussed it with +Reuven Lax >> > <[email protected]> and it appears to be quite difficult to do >> properly, or >> > even to define what it means at all, especially if you're using the >> > streaming inserts write method. So for now there is no workaround except >> > programmatically waiting for your whole pipeline to finish >> > (pipeline.run().waitUntilFinish()). >> > >> > On Fri, Sep 8, 2017 at 2:19 AM Chaim Turkel <[email protected]> wrote: >> > >> > > is there a way around this for now? >> > > how can i get a snapshot version? >> > > >> > > chaim >> > > >> > > On Tue, Sep 5, 2017 at 8:48 AM, Eugene Kirpichov >> > > <[email protected]> wrote: >> > > > Oh I see! Okay, this should be easy to fix. I'll take a look. >> > > > >> > > > On Mon, Sep 4, 2017 at 10:23 PM Chaim Turkel <[email protected]> >> wrote: >> > > > >> > > >> WriteResult does not support apply -> that is the problem >> > > >> >> > > >> On Tue, Sep 5, 2017 at 4:59 AM, Eugene Kirpichov >> > > >> <[email protected]> wrote: >> > > >> > Hi, >> > > >> > >> > > >> > Sorry for the delay. So sounds like you want to do something >> after >> > > >> writing >> > > >> > a window of data to BigQuery is complete. >> > > >> > I think this should be possible: expansion of BigQueryIO.write() >> > > returns >> > > >> a >> > > >> > WriteResult and you can apply other transforms to it. Have you >> tried >> > > >> that? >> > > >> > >> > > >> > On Sat, Aug 26, 2017 at 1:10 PM Chaim Turkel <[email protected]> >> > > wrote: >> > > >> > >> > > >> >> I have documents from a mongo db that i need to migrate to >> > bigquery. >> > > >> >> Since it is mongodb i do not know they schema ahead of time, so >> i >> > > have >> > > >> >> two pipelines, one to run over the documents and update the >> > bigquery >> > > >> >> schema, then wait a few minutes (i can take for bigquery to be >> able >> > > to >> > > >> >> use the new schema) then with the other pipline copy all the >> > > >> >> documents. >> > > >> >> To know as to where i got with the different piplines i have a >> > status >> > > >> >> table so that at the start i know from where to continue. >> > > >> >> So i need the option to update the status table with the >> success of >> > > >> >> the copy and some time value of the last copied document >> > > >> >> >> > > >> >> >> > > >> >> chaim >> > > >> >> >> > > >> >> On Fri, Aug 25, 2017 at 6:53 PM, Eugene Kirpichov >> > > >> >> <[email protected]> wrote: >> > > >> >> > I'd like to know more about your both use cases, can you >> > clarify? I >> > > >> think >> > > >> >> > making sinks output something that can be waited on by another >> > > >> pipeline >> > > >> >> > step is a reasonable request, but more details would help >> refine >> > > this >> > > >> >> > suggestion. >> > > >> >> > >> > > >> >> > On Fri, Aug 25, 2017, 8:46 AM Chamikara Jayalath < >> > > >> [email protected]> >> > > >> >> > wrote: >> > > >> >> > >> > > >> >> >> Can you do this from the program that runs the Beam job, >> after >> > > job is >> > > >> >> >> complete (you might have to use a blocking runner or poll for >> > the >> > > >> >> status of >> > > >> >> >> the job) ? >> > > >> >> >> >> > > >> >> >> - Cham >> > > >> >> >> >> > > >> >> >> On Fri, Aug 25, 2017 at 8:44 AM Steve Niemitz < >> > > [email protected]> >> > > >> >> wrote: >> > > >> >> >> >> > > >> >> >> > I also have a similar use case (but with BigTable) that I >> feel >> > > >> like I >> > > >> >> had >> > > >> >> >> > to hack up to make work. It'd be great to hear if there >> is a >> > > way >> > > >> to >> > > >> >> do >> > > >> >> >> > something like this already, or if there are plans in the >> > > future. >> > > >> >> >> > >> > > >> >> >> > On Fri, Aug 25, 2017 at 9:46 AM, Chaim Turkel < >> > [email protected] >> > > > >> > > >> >> wrote: >> > > >> >> >> > >> > > >> >> >> > > Hi, >> > > >> >> >> > > I have a few piplines that are an ETL from different >> > > systems to >> > > >> >> >> > bigquery. >> > > >> >> >> > > I would like to write the status of the ETL after all >> > records >> > > >> have >> > > >> >> >> > > been updated to the bigquery. >> > > >> >> >> > > The problem is that writing to bigquery is a sink and you >> > > cannot >> > > >> >> have >> > > >> >> >> > > any other steps after the sink. >> > > >> >> >> > > I tried a sideoutput, but this is called in no >> correlation >> > to >> > > the >> > > >> >> >> > > writing to bigquery, so i don't know if it succeeded or >> > > failed. >> > > >> >> >> > > >> > > >> >> >> > > >> > > >> >> >> > > any ideas? >> > > >> >> >> > > chaim >> > > >> >> >> > > >> > > >> >> >> > >> > > >> >> >> >> > > >> >> >> > > >> >> > > >> > >> >
