Re: Go SDK: How are re-starts handled?
BEAM-4665 created.
On Wed, Jun 27, 2018 at 4:32 PM Ismaël Mejía wrote:
> Eduardo can you please create a JIra on the Go SDK to track this issue.
> Thanks.
>
> On Mon, Jun 25, 2018 at 10:22 PM Lukasz Cwik wrote:
>
>> Ah, sorry for the confusion. The SDK is meant to handle that for you as I
>> described. You'll want to use the fact that the 409 was returned until that
>> is implemented within the Go SDK.
>>
>> On Mon, Jun 25, 2018 at 1:13 PM eduardo.mora...@gmail.com <
>> eduardo.mora...@gmail.com> wrote:
>>
>>> Nope. It returns an error.
>>>
>>> 2018/06/25 20:10:46 Failed to execute job: googleapi: Error 409:
>>> (acbae89877e14d87): The workflow could not be created. Causes:
>>> (590297f494c27357): There is already an active job named xxx-yyy_zzz. If
>>> you want to submit a second job, try again by setting a different name.,
>>> alreadyExists
>>>
>>> On 2018/06/25 16:18:55, Lukasz Cwik wrote:
>>> > It should be that beamx.Run won't return an error if a job already
>>> exists
>>> > with the same job name.
>>> >
>>> > On Mon, Jun 25, 2018 at 9:15 AM eduardo.mora...@gmail.com <
>>> > eduardo.mora...@gmail.com> wrote:
>>> >
>>> > > I am sorry. I am not expressing myself correctly. Let me do it
>>> though code:
>>> > >
>>> > > if err := beamx.Run(ctx, pipe); err != nil {
>>> > > // How do I know 'err' is the result of a pipeline already
>>> running, as
>>> > > opposed to some
>>> > > // other problem that may need special attention.
>>> > > }
>>> > >
>>> > > On 2018/06/22 23:10:13, Lukasz Cwik wrote:
>>> > > > The job name is a user chosen value[1]. If you don't specify
>>> something, a
>>> > > > job name is generated for you automatically[2].
>>> > > >
>>> > > > 1:
>>> > > >
>>> > >
>>> https://github.com/apache/beam/blob/c1927cd339c57125e29a651e614fb5105abf6d33/sdks/go/pkg/beam/options/jobopts/options.go#L38
>>> > > > 2:
>>> > > >
>>> > >
>>> https://github.com/apache/beam/blob/c1927cd339c57125e29a651e614fb5105abf6d33/sdks/go/pkg/beam/options/jobopts/options.go#L71
>>> > > >
>>> > > > On Fri, Jun 22, 2018 at 3:28 PM eduardo.mora...@gmail.com <
>>> > > > eduardo.mora...@gmail.com> wrote:
>>> > > >
>>> > > > >
>>> > > > >
>>> > > > > On 2018/06/22 21:35:29, Lukasz Cwik wrote:
>>> > > > > > There can only be one pipeline in Dataflow with the same job
>>> name so
>>> > > if
>>> > > > > you
>>> > > > > > attempt to submit another job with the same job name you'll
>>> get back
>>> > > an
>>> > > > > > identifier for the currently executing pipeline.
>>> > > > >
>>> > > > > But beam.Run() only returns an error. How do I get the job name
>>> back?
>>> > > > > My guess is that I have to use a different API (
>>> > > > > https://godoc.org/google.golang.org/api/dataflow/v1b3). Is that
>>> the
>>> > > > > correct way to detect job name collisions?
>>> > > > >
>>> > > > > Thanks again.
>>> > > > >
>>> > > > > > On Fri, Jun 22, 2018 at 2:27 PM eduardo.mora...@gmail.com <
>>> > > > > > eduardo.mora...@gmail.com> wrote:
>>> > > > > >
>>> > > > > > > If I have a k8s process launching dataflow pipelines, what
>>> happens
>>> > > when
>>> > > > > > > the process is restarted? Can Apache Beam detect a running
>>> > > pipeline and
>>> > > > > > > join accordingly? or will the pipeline be duplicated?
>>> > > > > > >
>>> > > > > > > Thanks in advance.
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>
Re: Go SDK: How are re-starts handled?
Eduardo can you please create a JIra on the Go SDK to track this issue.
Thanks.
On Mon, Jun 25, 2018 at 10:22 PM Lukasz Cwik wrote:
> Ah, sorry for the confusion. The SDK is meant to handle that for you as I
> described. You'll want to use the fact that the 409 was returned until that
> is implemented within the Go SDK.
>
> On Mon, Jun 25, 2018 at 1:13 PM eduardo.mora...@gmail.com <
> eduardo.mora...@gmail.com> wrote:
>
>> Nope. It returns an error.
>>
>> 2018/06/25 20:10:46 Failed to execute job: googleapi: Error 409:
>> (acbae89877e14d87): The workflow could not be created. Causes:
>> (590297f494c27357): There is already an active job named xxx-yyy_zzz. If
>> you want to submit a second job, try again by setting a different name.,
>> alreadyExists
>>
>> On 2018/06/25 16:18:55, Lukasz Cwik wrote:
>> > It should be that beamx.Run won't return an error if a job already
>> exists
>> > with the same job name.
>> >
>> > On Mon, Jun 25, 2018 at 9:15 AM eduardo.mora...@gmail.com <
>> > eduardo.mora...@gmail.com> wrote:
>> >
>> > > I am sorry. I am not expressing myself correctly. Let me do it though
>> code:
>> > >
>> > > if err := beamx.Run(ctx, pipe); err != nil {
>> > > // How do I know 'err' is the result of a pipeline already running,
>> as
>> > > opposed to some
>> > > // other problem that may need special attention.
>> > > }
>> > >
>> > > On 2018/06/22 23:10:13, Lukasz Cwik wrote:
>> > > > The job name is a user chosen value[1]. If you don't specify
>> something, a
>> > > > job name is generated for you automatically[2].
>> > > >
>> > > > 1:
>> > > >
>> > >
>> https://github.com/apache/beam/blob/c1927cd339c57125e29a651e614fb5105abf6d33/sdks/go/pkg/beam/options/jobopts/options.go#L38
>> > > > 2:
>> > > >
>> > >
>> https://github.com/apache/beam/blob/c1927cd339c57125e29a651e614fb5105abf6d33/sdks/go/pkg/beam/options/jobopts/options.go#L71
>> > > >
>> > > > On Fri, Jun 22, 2018 at 3:28 PM eduardo.mora...@gmail.com <
>> > > > eduardo.mora...@gmail.com> wrote:
>> > > >
>> > > > >
>> > > > >
>> > > > > On 2018/06/22 21:35:29, Lukasz Cwik wrote:
>> > > > > > There can only be one pipeline in Dataflow with the same job
>> name so
>> > > if
>> > > > > you
>> > > > > > attempt to submit another job with the same job name you'll get
>> back
>> > > an
>> > > > > > identifier for the currently executing pipeline.
>> > > > >
>> > > > > But beam.Run() only returns an error. How do I get the job name
>> back?
>> > > > > My guess is that I have to use a different API (
>> > > > > https://godoc.org/google.golang.org/api/dataflow/v1b3). Is that
>> the
>> > > > > correct way to detect job name collisions?
>> > > > >
>> > > > > Thanks again.
>> > > > >
>> > > > > > On Fri, Jun 22, 2018 at 2:27 PM eduardo.mora...@gmail.com <
>> > > > > > eduardo.mora...@gmail.com> wrote:
>> > > > > >
>> > > > > > > If I have a k8s process launching dataflow pipelines, what
>> happens
>> > > when
>> > > > > > > the process is restarted? Can Apache Beam detect a running
>> > > pipeline and
>> > > > > > > join accordingly? or will the pipeline be duplicated?
>> > > > > > >
>> > > > > > > Thanks in advance.
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: Go SDK: How are re-starts handled?
Ah, sorry for the confusion. The SDK is meant to handle that for you as I
described. You'll want to use the fact that the 409 was returned until that
is implemented within the Go SDK.
On Mon, Jun 25, 2018 at 1:13 PM eduardo.mora...@gmail.com <
eduardo.mora...@gmail.com> wrote:
> Nope. It returns an error.
>
> 2018/06/25 20:10:46 Failed to execute job: googleapi: Error 409:
> (acbae89877e14d87): The workflow could not be created. Causes:
> (590297f494c27357): There is already an active job named xxx-yyy_zzz. If
> you want to submit a second job, try again by setting a different name.,
> alreadyExists
>
> On 2018/06/25 16:18:55, Lukasz Cwik wrote:
> > It should be that beamx.Run won't return an error if a job already exists
> > with the same job name.
> >
> > On Mon, Jun 25, 2018 at 9:15 AM eduardo.mora...@gmail.com <
> > eduardo.mora...@gmail.com> wrote:
> >
> > > I am sorry. I am not expressing myself correctly. Let me do it though
> code:
> > >
> > > if err := beamx.Run(ctx, pipe); err != nil {
> > > // How do I know 'err' is the result of a pipeline already running,
> as
> > > opposed to some
> > > // other problem that may need special attention.
> > > }
> > >
> > > On 2018/06/22 23:10:13, Lukasz Cwik wrote:
> > > > The job name is a user chosen value[1]. If you don't specify
> something, a
> > > > job name is generated for you automatically[2].
> > > >
> > > > 1:
> > > >
> > >
> https://github.com/apache/beam/blob/c1927cd339c57125e29a651e614fb5105abf6d33/sdks/go/pkg/beam/options/jobopts/options.go#L38
> > > > 2:
> > > >
> > >
> https://github.com/apache/beam/blob/c1927cd339c57125e29a651e614fb5105abf6d33/sdks/go/pkg/beam/options/jobopts/options.go#L71
> > > >
> > > > On Fri, Jun 22, 2018 at 3:28 PM eduardo.mora...@gmail.com <
> > > > eduardo.mora...@gmail.com> wrote:
> > > >
> > > > >
> > > > >
> > > > > On 2018/06/22 21:35:29, Lukasz Cwik wrote:
> > > > > > There can only be one pipeline in Dataflow with the same job
> name so
> > > if
> > > > > you
> > > > > > attempt to submit another job with the same job name you'll get
> back
> > > an
> > > > > > identifier for the currently executing pipeline.
> > > > >
> > > > > But beam.Run() only returns an error. How do I get the job name
> back?
> > > > > My guess is that I have to use a different API (
> > > > > https://godoc.org/google.golang.org/api/dataflow/v1b3). Is that
> the
> > > > > correct way to detect job name collisions?
> > > > >
> > > > > Thanks again.
> > > > >
> > > > > > On Fri, Jun 22, 2018 at 2:27 PM eduardo.mora...@gmail.com <
> > > > > > eduardo.mora...@gmail.com> wrote:
> > > > > >
> > > > > > > If I have a k8s process launching dataflow pipelines, what
> happens
> > > when
> > > > > > > the process is restarted? Can Apache Beam detect a running
> > > pipeline and
> > > > > > > join accordingly? or will the pipeline be duplicated?
> > > > > > >
> > > > > > > Thanks in advance.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Go SDK: How are re-starts handled?
Nope. It returns an error.
2018/06/25 20:10:46 Failed to execute job: googleapi: Error 409:
(acbae89877e14d87): The workflow could not be created. Causes:
(590297f494c27357): There is already an active job named xxx-yyy_zzz. If you
want to submit a second job, try again by setting a different name.,
alreadyExists
On 2018/06/25 16:18:55, Lukasz Cwik wrote:
> It should be that beamx.Run won't return an error if a job already exists
> with the same job name.
>
> On Mon, Jun 25, 2018 at 9:15 AM eduardo.mora...@gmail.com <
> eduardo.mora...@gmail.com> wrote:
>
> > I am sorry. I am not expressing myself correctly. Let me do it though code:
> >
> > if err := beamx.Run(ctx, pipe); err != nil {
> > // How do I know 'err' is the result of a pipeline already running, as
> > opposed to some
> > // other problem that may need special attention.
> > }
> >
> > On 2018/06/22 23:10:13, Lukasz Cwik wrote:
> > > The job name is a user chosen value[1]. If you don't specify something, a
> > > job name is generated for you automatically[2].
> > >
> > > 1:
> > >
> > https://github.com/apache/beam/blob/c1927cd339c57125e29a651e614fb5105abf6d33/sdks/go/pkg/beam/options/jobopts/options.go#L38
> > > 2:
> > >
> > https://github.com/apache/beam/blob/c1927cd339c57125e29a651e614fb5105abf6d33/sdks/go/pkg/beam/options/jobopts/options.go#L71
> > >
> > > On Fri, Jun 22, 2018 at 3:28 PM eduardo.mora...@gmail.com <
> > > eduardo.mora...@gmail.com> wrote:
> > >
> > > >
> > > >
> > > > On 2018/06/22 21:35:29, Lukasz Cwik wrote:
> > > > > There can only be one pipeline in Dataflow with the same job name so
> > if
> > > > you
> > > > > attempt to submit another job with the same job name you'll get back
> > an
> > > > > identifier for the currently executing pipeline.
> > > >
> > > > But beam.Run() only returns an error. How do I get the job name back?
> > > > My guess is that I have to use a different API (
> > > > https://godoc.org/google.golang.org/api/dataflow/v1b3). Is that the
> > > > correct way to detect job name collisions?
> > > >
> > > > Thanks again.
> > > >
> > > > > On Fri, Jun 22, 2018 at 2:27 PM eduardo.mora...@gmail.com <
> > > > > eduardo.mora...@gmail.com> wrote:
> > > > >
> > > > > > If I have a k8s process launching dataflow pipelines, what happens
> > when
> > > > > > the process is restarted? Can Apache Beam detect a running
> > pipeline and
> > > > > > join accordingly? or will the pipeline be duplicated?
> > > > > >
> > > > > > Thanks in advance.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Go SDK: How are re-starts handled?
It should be that beamx.Run won't return an error if a job already exists
with the same job name.
On Mon, Jun 25, 2018 at 9:15 AM eduardo.mora...@gmail.com <
eduardo.mora...@gmail.com> wrote:
> I am sorry. I am not expressing myself correctly. Let me do it though code:
>
> if err := beamx.Run(ctx, pipe); err != nil {
> // How do I know 'err' is the result of a pipeline already running, as
> opposed to some
> // other problem that may need special attention.
> }
>
> On 2018/06/22 23:10:13, Lukasz Cwik wrote:
> > The job name is a user chosen value[1]. If you don't specify something, a
> > job name is generated for you automatically[2].
> >
> > 1:
> >
> https://github.com/apache/beam/blob/c1927cd339c57125e29a651e614fb5105abf6d33/sdks/go/pkg/beam/options/jobopts/options.go#L38
> > 2:
> >
> https://github.com/apache/beam/blob/c1927cd339c57125e29a651e614fb5105abf6d33/sdks/go/pkg/beam/options/jobopts/options.go#L71
> >
> > On Fri, Jun 22, 2018 at 3:28 PM eduardo.mora...@gmail.com <
> > eduardo.mora...@gmail.com> wrote:
> >
> > >
> > >
> > > On 2018/06/22 21:35:29, Lukasz Cwik wrote:
> > > > There can only be one pipeline in Dataflow with the same job name so
> if
> > > you
> > > > attempt to submit another job with the same job name you'll get back
> an
> > > > identifier for the currently executing pipeline.
> > >
> > > But beam.Run() only returns an error. How do I get the job name back?
> > > My guess is that I have to use a different API (
> > > https://godoc.org/google.golang.org/api/dataflow/v1b3). Is that the
> > > correct way to detect job name collisions?
> > >
> > > Thanks again.
> > >
> > > > On Fri, Jun 22, 2018 at 2:27 PM eduardo.mora...@gmail.com <
> > > > eduardo.mora...@gmail.com> wrote:
> > > >
> > > > > If I have a k8s process launching dataflow pipelines, what happens
> when
> > > > > the process is restarted? Can Apache Beam detect a running
> pipeline and
> > > > > join accordingly? or will the pipeline be duplicated?
> > > > >
> > > > > Thanks in advance.
> > > > >
> > > >
> > >
> >
>
Re: Go SDK: How are re-starts handled?
I am sorry. I am not expressing myself correctly. Let me do it though code:
if err := beamx.Run(ctx, pipe); err != nil {
// How do I know 'err' is the result of a pipeline already running, as
opposed to some
// other problem that may need special attention.
}
On 2018/06/22 23:10:13, Lukasz Cwik wrote:
> The job name is a user chosen value[1]. If you don't specify something, a
> job name is generated for you automatically[2].
>
> 1:
> https://github.com/apache/beam/blob/c1927cd339c57125e29a651e614fb5105abf6d33/sdks/go/pkg/beam/options/jobopts/options.go#L38
> 2:
> https://github.com/apache/beam/blob/c1927cd339c57125e29a651e614fb5105abf6d33/sdks/go/pkg/beam/options/jobopts/options.go#L71
>
> On Fri, Jun 22, 2018 at 3:28 PM eduardo.mora...@gmail.com <
> eduardo.mora...@gmail.com> wrote:
>
> >
> >
> > On 2018/06/22 21:35:29, Lukasz Cwik wrote:
> > > There can only be one pipeline in Dataflow with the same job name so if
> > you
> > > attempt to submit another job with the same job name you'll get back an
> > > identifier for the currently executing pipeline.
> >
> > But beam.Run() only returns an error. How do I get the job name back?
> > My guess is that I have to use a different API (
> > https://godoc.org/google.golang.org/api/dataflow/v1b3). Is that the
> > correct way to detect job name collisions?
> >
> > Thanks again.
> >
> > > On Fri, Jun 22, 2018 at 2:27 PM eduardo.mora...@gmail.com <
> > > eduardo.mora...@gmail.com> wrote:
> > >
> > > > If I have a k8s process launching dataflow pipelines, what happens when
> > > > the process is restarted? Can Apache Beam detect a running pipeline and
> > > > join accordingly? or will the pipeline be duplicated?
> > > >
> > > > Thanks in advance.
> > > >
> > >
> >
>
Re: Go SDK: How are re-starts handled?
The job name is a user chosen value[1]. If you don't specify something, a job name is generated for you automatically[2]. 1: https://github.com/apache/beam/blob/c1927cd339c57125e29a651e614fb5105abf6d33/sdks/go/pkg/beam/options/jobopts/options.go#L38 2: https://github.com/apache/beam/blob/c1927cd339c57125e29a651e614fb5105abf6d33/sdks/go/pkg/beam/options/jobopts/options.go#L71 On Fri, Jun 22, 2018 at 3:28 PM eduardo.mora...@gmail.com < eduardo.mora...@gmail.com> wrote: > > > On 2018/06/22 21:35:29, Lukasz Cwik wrote: > > There can only be one pipeline in Dataflow with the same job name so if > you > > attempt to submit another job with the same job name you'll get back an > > identifier for the currently executing pipeline. > > But beam.Run() only returns an error. How do I get the job name back? > My guess is that I have to use a different API ( > https://godoc.org/google.golang.org/api/dataflow/v1b3). Is that the > correct way to detect job name collisions? > > Thanks again. > > > On Fri, Jun 22, 2018 at 2:27 PM eduardo.mora...@gmail.com < > > eduardo.mora...@gmail.com> wrote: > > > > > If I have a k8s process launching dataflow pipelines, what happens when > > > the process is restarted? Can Apache Beam detect a running pipeline and > > > join accordingly? or will the pipeline be duplicated? > > > > > > Thanks in advance. > > > > > >
Re: Go SDK: How are re-starts handled?
On 2018/06/22 21:35:29, Lukasz Cwik wrote: > There can only be one pipeline in Dataflow with the same job name so if you > attempt to submit another job with the same job name you'll get back an > identifier for the currently executing pipeline. But beam.Run() only returns an error. How do I get the job name back? My guess is that I have to use a different API (https://godoc.org/google.golang.org/api/dataflow/v1b3). Is that the correct way to detect job name collisions? Thanks again. > On Fri, Jun 22, 2018 at 2:27 PM eduardo.mora...@gmail.com < > eduardo.mora...@gmail.com> wrote: > > > If I have a k8s process launching dataflow pipelines, what happens when > > the process is restarted? Can Apache Beam detect a running pipeline and > > join accordingly? or will the pipeline be duplicated? > > > > Thanks in advance. > > >
Re: Go SDK: How are re-starts handled?
There can only be one pipeline in Dataflow with the same job name so if you attempt to submit another job with the same job name you'll get back an identifier for the currently executing pipeline. On Fri, Jun 22, 2018 at 2:27 PM eduardo.mora...@gmail.com < eduardo.mora...@gmail.com> wrote: > If I have a k8s process launching dataflow pipelines, what happens when > the process is restarted? Can Apache Beam detect a running pipeline and > join accordingly? or will the pipeline be duplicated? > > Thanks in advance. >
