Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

Hello Again! So, the External stuff is how the "built in" implementation in runners is being triggered. That ends up adding a Transform in the raw Pipeline proto [0] that has the PubSub Write URN [1] which is how the Runner (Dataflow) knows that it should be doing something more for this. As

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

Hi folks, Finding it a little hard to figure out the starting point. If someone from the community can pair with me on this for 30-45 mins, that would be great. I have read the code in the pubsubio package and drilled down into the `beam.External` function as well but couldn't make much sense. T

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

One solution would be to make the substitution only at Pipeline Submission time, when we know if the job is being submitted to Batch Dataflow or not, and either 1. Fail with a clear error message, or 2. Substitute the external transform with the batch publish DoFn. This is closer to how Java and P

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

I've added a comment to the StackOverflow question . I guess the suggested approach there is the workaround for now, unfortunately. You have to create a new client in the setup

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

Hi Shivam, Thanks a lot for your response. Yes it is a batch pipeline. My task is to read a big query table, process the data and publish the Rows as a PubSub message. Regards, Ashok On Mon, 6 Feb 2023 at 10:52 pm, Shivam Singhal wrote: > Hey Ashok KS, > > Is this a batch pipeline? > > On Mon,

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

I will be picking the issue up once the maintainers have triaged the issue. On Mon, 6 Feb 2023 at 17:43, Shivam Singhal wrote: > Not sure if there is any solution other than fixing the Go pubsubio > package. > > On Mon, 6 Feb 2023 at 17:41, Ashok KS wrote: > >> Yes, that is where Iam getting st

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

Not sure if there is any solution other than fixing the Go pubsubio package. On Mon, 6 Feb 2023 at 17:41, Ashok KS wrote: > Yes, that is where Iam getting stuck. I wrote the complete pipeline in > Python which reads from the BQ table and published it as a PubSub message. > I'm able to force it a

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

It depends on the input source: it will decide if your pipeline is a streaming or a batch pipeline. Since you are querying over a BQ table, the input is finite and in result, your pipeline is a batch pipeline. I am not sure there is a straightforward way where you can convert this pipeline into a

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

The issue is not yet verified by the maintainers but I think the pubsubio connector's Write method doesn't work in Batch pipelines. But I am pretty sure that pubsubio Write doesn't work for Batch Pipelines because it's mentioned in the code comments. Check the below issue for the details: https://

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

Hey Ashok KS, Is this a batch pipeline? On Mon, 6 Feb 2023 at 09:27, Ashok KS wrote: > Hi All, > > Just sending a reminder in case anyone could help. I haven't received any > response to my issue. > > Regards, > Ashok > > On Fri, Feb 3, 2023 at 12:23 AM Ashok KS wrote: > >> Hi All, >> >> I'm n

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

Hi All, Just sending a reminder in case anyone could help. I haven't received any response to my issue. Regards, Ashok On Fri, Feb 3, 2023 at 12:23 AM Ashok KS wrote: > Hi All, > > I'm new to using Apache Beam using Go. > > pubsubio.Write(scope, "project", "topic", ppMessages) > When I try to