I'm not 100% sure as I haven't tried it, but, Combining comes to mind as a possible way of doing this, assuming your data is finite
https://beam.apache.org/documentation/programming-guide/#transforms-combine You could take the PCollection result of 2 and simultaneously apply the Write and the Combine, using the singular result of the Combine to trigger the remaining steps Hope that helps, I'm still learning -Gwilym On 11 June 2017 at 16:50, Morand, Sebastien <[email protected]> wrote: > No, I can't, the pipeline is created within a cron, which is limited to 10 > minutes. > > *Sébastien MORAND* > Team Lead Solution Architect > Technology & Operations / Digital Factory > Veolia - Group Information Systems & Technology (IS&T) > Cell.: +33 7 52 66 20 81 / Direct: +33 1 85 57 71 08 > <+33%201%2085%2057%2071%2008> > Bureau 0144C (Ouest) > 30, rue Madeleine-Vionnet - 93300 Aubervilliers, France > *www.veolia.com <http://www.veolia.com>* > <http://www.veolia.com> > <https://www.facebook.com/veoliaenvironment/> > <https://www.youtube.com/user/veoliaenvironnement> > <https://www.linkedin.com/company/veolia-environnement> > <https://twitter.com/veolia> > > On 11 June 2017 at 18:21, Eugene Kirpichov <[email protected]> wrote: > >> Hmm can you simply do this in your main program after the pipeline >> finishes? >> >> p.run().waitUntilFinish(); >> ... Send report ... >> >> On Sun, Jun 11, 2017, 1:50 AM Morand, Sebastien < >> [email protected]> wrote: >> >>> Yes this use case can be treated by using parallel operation. >>> >>> I have a 2nd one, I would like to send a report at the end of the >>> pipeline when the last line has been written in bigquery: number of lines >>> treated, number of lines ignored (from another part of the pipeline using >>> graph as you described), number of files at the begining, and so on. >>> >>> This report could be: >>> >>> 1. Write a pub/sub >>> 2. Send an email >>> 3. Call an url with parameters >>> >>> Is this possible? >>> >>> Regards, >>> >>> >>> >>> *Sébastien MORAND* >>> Team Lead Solution Architect >>> Technology & Operations / Digital Factory >>> Veolia - Group Information Systems & Technology (IS&T) >>> Cell.: +33 7 52 66 20 81 / Direct: +33 1 85 57 71 08 >>> <+33%201%2085%2057%2071%2008> >>> Bureau 0144C (Ouest) >>> 30, rue Madeleine-Vionnet - 93300 Aubervilliers, France >>> *www.veolia.com <http://www.veolia.com>* >>> <http://www.veolia.com> >>> <https://www.facebook.com/veoliaenvironment/> >>> <https://www.youtube.com/user/veoliaenvironnement> >>> <https://www.linkedin.com/company/veolia-environnement> >>> <https://twitter.com/veolia> >>> >>> On 11 June 2017 at 04:14, Eugene Kirpichov <[email protected]> wrote: >>> >>>> Hi! >>>> It sounds like you want to write data to BigQuery and then load the >>>> same data back from BigQuery? Why? I'm particularly confused by your >>>> comment "nothing left in the PCollection" - writing a collection to >>>> BigQuery doesn't remove data from the collection, a PCollection is just a >>>> logical description of a dataset, not a mutable container. Transforms are >>>> like mathematical functions - they don't change their inputs, they only >>>> compute their outputs. >>>> >>>> Perhaps that you're assuming that Beam pipelines can only be a strict >>>> linear sequence of transforms? That is not the case - pipelines are an >>>> arbitrary graph, you can use a collection multiple times, i.e. apply >>>> multiple transforms to it. E.g. you can both write the collection to >>>> bigquery (step 3) and apply some other transform to the same collection >>>> (step 5). >>>> >>>> Assuming you use Java: >>>> PCollection<Foo> foos = p.apply(TextIO.read().from(...)).apply(...some >>>> transform...); >>>> foos.apply(BigQueryIO.write().to(...)); >>>> PCollection<Bar> bars = foos.apply(...some other transform...); >>>> bars.apply(BigQueryIO.write().to(...)); >>>> >>>> Let me know if this helps. >>>> >>>> On Sat, Jun 10, 2017 at 3:42 PM Morand, Sebastien < >>>> [email protected]> wrote: >>>> >>>>> Hi, >>>>> >>>>> Is there any way to add some step after a Write, because Write return >>>>> un PDone, so I can't do anything, but I would like actually do something. >>>>> >>>>> Example : >>>>> >>>>> 1. Load data from gcs >>>>> 2. Some transform >>>>> 3. Write data into bigquery >>>>> => Nothing left in the pcollection, but when 3 is over => >>>>> 4. Load data from bigquery >>>>> 5. Some other transform >>>>> 6. Write data into bigquery >>>>> >>>>> Any way to do that? >>>>> >>>>> Thanks, >>>>> >>>>> *Sébastien MORAND* >>>>> Team Lead Solution Architect >>>>> Technology & Operations / Digital Factory >>>>> Veolia - Group Information Systems & Technology (IS&T) >>>>> Cell.: +33 7 52 66 20 81 / Direct: +33 1 85 57 71 08 >>>>> <+33%201%2085%2057%2071%2008> >>>>> Bureau 0144C (Ouest) >>>>> 30, rue Madeleine-Vionnet - 93300 Aubervilliers, France >>>>> *www.veolia.com <http://www.veolia.com>* >>>>> <http://www.veolia.com> >>>>> <https://www.facebook.com/veoliaenvironment/> >>>>> <https://www.youtube.com/user/veoliaenvironnement> >>>>> <https://www.linkedin.com/company/veolia-environnement> >>>>> <https://twitter.com/veolia> >>>>> >>>>> >>>>> ------------------------------------------------------------ >>>>> -------------------------------- >>>>> This e-mail transmission (message and any attached files) may contain >>>>> information that is proprietary, privileged and/or confidential to Veolia >>>>> Environnement and/or its affiliates and is intended exclusively for the >>>>> person(s) to whom it is addressed. If you are not the intended recipient, >>>>> please notify the sender by return e-mail and delete all copies of this >>>>> e-mail, including all attachments. Unless expressly authorized, any use, >>>>> disclosure, publication, retransmission or dissemination of this e-mail >>>>> and/or of its attachments is strictly prohibited. >>>>> >>>>> Ce message electronique et ses fichiers attaches sont strictement >>>>> confidentiels et peuvent contenir des elements dont Veolia Environnement >>>>> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc >>>>> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce >>>>> message par erreur, merci de le retourner a son emetteur et de le detruire >>>>> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la >>>>> publication, la distribution, ou la reproduction non expressement >>>>> autorisees de ce message et de ses pieces attachees sont interdites. >>>>> ------------------------------------------------------------ >>>>> -------------------------------- >>>>> >>>> >>> >>> >>> ------------------------------------------------------------ >>> -------------------------------- >>> This e-mail transmission (message and any attached files) may contain >>> information that is proprietary, privileged and/or confidential to Veolia >>> Environnement and/or its affiliates and is intended exclusively for the >>> person(s) to whom it is addressed. If you are not the intended recipient, >>> please notify the sender by return e-mail and delete all copies of this >>> e-mail, including all attachments. Unless expressly authorized, any use, >>> disclosure, publication, retransmission or dissemination of this e-mail >>> and/or of its attachments is strictly prohibited. >>> >>> Ce message electronique et ses fichiers attaches sont strictement >>> confidentiels et peuvent contenir des elements dont Veolia Environnement >>> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc >>> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce >>> message par erreur, merci de le retourner a son emetteur et de le detruire >>> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la >>> publication, la distribution, ou la reproduction non expressement >>> autorisees de ce message et de ses pieces attachees sont interdites. >>> ------------------------------------------------------------ >>> -------------------------------- >>> >> > > > ------------------------------------------------------------ > -------------------------------- > This e-mail transmission (message and any attached files) may contain > information that is proprietary, privileged and/or confidential to Veolia > Environnement and/or its affiliates and is intended exclusively for the > person(s) to whom it is addressed. If you are not the intended recipient, > please notify the sender by return e-mail and delete all copies of this > e-mail, including all attachments. Unless expressly authorized, any use, > disclosure, publication, retransmission or dissemination of this e-mail > and/or of its attachments is strictly prohibited. > > Ce message electronique et ses fichiers attaches sont strictement > confidentiels et peuvent contenir des elements dont Veolia Environnement > et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc > destines a l'usage de leurs seuls destinataires. Si vous avez recu ce > message par erreur, merci de le retourner a son emetteur et de le detruire > ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la > publication, la distribution, ou la reproduction non expressement > autorisees de ce message et de ses pieces attachees sont interdites. > ------------------------------------------------------------ > -------------------------------- >
