This is what I was referencing: https://github.com/googleapis/google-api-java-client-services/tree/master/clients/google-api-services-dataflow/v1b3
On Mon, Oct 12, 2020 at 2:23 PM Peter Littig <plit...@nianticlabs.com> wrote: > Thanks for the replies, Lukasz and Steve! > > Steve: do you have a link to the google client api wrappers (I'm not sure > if I know what they are.) > > Thank you! > > On Mon, Oct 12, 2020 at 11:04 AM Steve Niemitz <sniem...@apache.org> > wrote: > >> We use the Dataflow API [1] directly, via the google api client wrappers >> (both python and java), pretty extensively. It works well and doesn't >> require a dependency on beam. >> >> [1] https://cloud.google.com/dataflow/docs/reference/rest >> >> On Mon, Oct 12, 2020 at 1:56 PM Luke Cwik <lc...@google.com> wrote: >> >>> It is your best way to do this right now and this hasn't changed in a >>> while (region was added to project and job ids in the past 6 years). >>> >>> On Mon, Oct 12, 2020 at 10:53 AM Peter Littig <plit...@nianticlabs.com> >>> wrote: >>> >>>> Thanks for the reply, Kyle. >>>> >>>> The DataflowClient::getJob method uses a Dataflow instance that's >>>> provided at construction time (via >>>> DataflowPipelineOptions::getDataflowClient). If that Dataflow instance can >>>> be obtained from a minimal instance of the options (i.e., containing only >>>> the project ID and region) then it looks like everything should work. >>>> >>>> I suppose a secondary question here is whether or not this approach is >>>> the recommended way to solve my problem (but I don't know of any >>>> alternatives). >>>> >>>> On Mon, Oct 12, 2020 at 9:55 AM Kyle Weaver <kcwea...@google.com> >>>> wrote: >>>> >>>>> > I think the answer is to use a DataflowClient in the second service, >>>>> but creating one requires DataflowPipelineOptions. Are these options >>>>> supposed to be exactly the same as those used by the first service? Or do >>>>> only some of the fields have to be the same? >>>>> >>>>> Most options are not necessary for retrieving a job. In general, >>>>> Dataflow jobs can always be uniquely identified by the project, region and >>>>> job ID. >>>>> https://github.com/apache/beam/blob/ecedd3e654352f1b51ab2caae0fd4665403bd0eb/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowClient.java#L100 >>>>> >>>>> On Mon, Oct 12, 2020 at 9:31 AM Peter Littig <plit...@nianticlabs.com> >>>>> wrote: >>>>> >>>>>> Hello, Beam users! >>>>>> >>>>>> Suppose I want to build two (Java) services, one that launches >>>>>> (long-running) dataflow jobs, and the other that monitors the status of >>>>>> dataflow jobs. Within a single service, I could simply track a >>>>>> PipelineResult for each dataflow run and periodically call getState. How >>>>>> can I monitor job status like this from a second, independent service? >>>>>> >>>>>> I think the answer is to use a DataflowClient in the second service, >>>>>> but creating one requires DataflowPipelineOptions. Are these options >>>>>> supposed to be exactly the same as those used by the first service? Or do >>>>>> only some of the fields have to be the same? >>>>>> >>>>>> Or maybe there's a better alternative than DataflowClient? >>>>>> >>>>>> Thanks in advance! >>>>>> >>>>>> Peter >>>>>> >>>>>
