Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

[Shivam Singhal]

I will be picking the issue up once the maintainers have triaged the issue.

On Mon, 6 Feb 2023 at 17:43, Shivam Singhal 
wrote:

> Not sure if there is any solution other than fixing the Go pubsubio
> package.
>
> On Mon, 6 Feb 2023 at 17:41, Ashok KS  wrote:
>
>> Yes, that is where Iam getting stuck. I wrote the complete pipeline in
>> Python which reads from the BQ table and published it as a PubSub message.
>> I'm able to force it as a streaming application by passing --streaming=True
>> But for my project, they want it in Go so I had to rewrite the complete
>> logic in Go.
>> I did the same, but stuck at the last point of publishing it to PubSub.
>>
>> On Mon, Feb 6, 2023 at 11:07 PM Shivam Singhal <
>> shivamsinghal5...@gmail.com> wrote:
>>
>>> It depends on the input source: it will decide if your pipeline is a
>>> streaming or a batch pipeline.
>>>
>>> Since you are querying over a BQ table, the input is finite and in
>>> result, your pipeline is a batch pipeline.
>>> I am not sure there is a straightforward way where you can convert this
>>> pipeline into a streaming pipeline.
>>>
>>>
>>> On Mon, 6 Feb 2023 at 17:32, Ashok KS  wrote:
>>>
 Hi Shivam,

 Thanks for that. How can run the pipeline as a streaming pipeline?  In
 python I could just run the pipeline by passing —streaming=True in the
 command line, but I couldn’t find anything similar in Go.

 Any pointers would be appreciated.

 Regards,
 Ashok

 On Mon, 6 Feb 2023 at 10:59 pm, Shivam Singhal <
 shivamsinghal5...@gmail.com> wrote:

> The issue is not yet verified by the maintainers but I think the
> pubsubio connector's Write method doesn't work in Batch pipelines.
>
> But I am pretty sure that pubsubio Write doesn't work for Batch
> Pipelines because it's mentioned in the code comments. Check the below
> issue for the details:
> https://github.com/apache/beam/issues/25326
>
> On Mon, 6 Feb 2023 at 17:26, Ashok KS  wrote:
>
>> Hi Shivam,
>>
>> Thanks a lot for your response. Yes it is a batch pipeline. My task
>> is to read a big query table, process the data and publish the Rows as a
>> PubSub message.
>>
>> Regards,
>> Ashok
>>
>> On Mon, 6 Feb 2023 at 10:52 pm, Shivam Singhal <
>> shivamsinghal5...@gmail.com> wrote:
>>
>>> Hey Ashok KS,
>>>
>>> Is this a batch pipeline?
>>>
>>> On Mon, 6 Feb 2023 at 09:27, Ashok KS  wrote:
>>>
 Hi All,

 Just sending a reminder in case anyone could help. I haven't
 received any response to my issue.

 Regards,
 Ashok

 On Fri, Feb 3, 2023 at 12:23 AM Ashok KS 
 wrote:

> Hi All,
>
> I'm new to using Apache Beam using Go.
>
> pubsubio.Write(scope, "project", "topic", ppMessages)
> When I try to publish a message in a topic I get an error message
> "Could not find the sink for pubsub, Check that the sink library
> specifies alwayslink = 1
>
> I found a StackOverFlow post for the same issue but it doesn't
> solve the problem.
>
> Stackoverflow Link
> 
>
> Can someone please help?
>
> Regards,
> Ashok
>

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

[Shivam Singhal]

Not sure if there is any solution other than fixing the Go pubsubio package.

On Mon, 6 Feb 2023 at 17:41, Ashok KS  wrote:

> Yes, that is where Iam getting stuck. I wrote the complete pipeline in
> Python which reads from the BQ table and published it as a PubSub message.
> I'm able to force it as a streaming application by passing --streaming=True
> But for my project, they want it in Go so I had to rewrite the complete
> logic in Go.
> I did the same, but stuck at the last point of publishing it to PubSub.
>
> On Mon, Feb 6, 2023 at 11:07 PM Shivam Singhal <
> shivamsinghal5...@gmail.com> wrote:
>
>> It depends on the input source: it will decide if your pipeline is a
>> streaming or a batch pipeline.
>>
>> Since you are querying over a BQ table, the input is finite and in
>> result, your pipeline is a batch pipeline.
>> I am not sure there is a straightforward way where you can convert this
>> pipeline into a streaming pipeline.
>>
>>
>> On Mon, 6 Feb 2023 at 17:32, Ashok KS  wrote:
>>
>>> Hi Shivam,
>>>
>>> Thanks for that. How can run the pipeline as a streaming pipeline?  In
>>> python I could just run the pipeline by passing —streaming=True in the
>>> command line, but I couldn’t find anything similar in Go.
>>>
>>> Any pointers would be appreciated.
>>>
>>> Regards,
>>> Ashok
>>>
>>> On Mon, 6 Feb 2023 at 10:59 pm, Shivam Singhal <
>>> shivamsinghal5...@gmail.com> wrote:
>>>
 The issue is not yet verified by the maintainers but I think the
 pubsubio connector's Write method doesn't work in Batch pipelines.

 But I am pretty sure that pubsubio Write doesn't work for Batch
 Pipelines because it's mentioned in the code comments. Check the below
 issue for the details:
 https://github.com/apache/beam/issues/25326

 On Mon, 6 Feb 2023 at 17:26, Ashok KS  wrote:

> Hi Shivam,
>
> Thanks a lot for your response. Yes it is a batch pipeline. My task is
> to read a big query table, process the data and publish the Rows as a
> PubSub message.
>
> Regards,
> Ashok
>
> On Mon, 6 Feb 2023 at 10:52 pm, Shivam Singhal <
> shivamsinghal5...@gmail.com> wrote:
>
>> Hey Ashok KS,
>>
>> Is this a batch pipeline?
>>
>> On Mon, 6 Feb 2023 at 09:27, Ashok KS  wrote:
>>
>>> Hi All,
>>>
>>> Just sending a reminder in case anyone could help. I haven't
>>> received any response to my issue.
>>>
>>> Regards,
>>> Ashok
>>>
>>> On Fri, Feb 3, 2023 at 12:23 AM Ashok KS 
>>> wrote:
>>>
 Hi All,

 I'm new to using Apache Beam using Go.

 pubsubio.Write(scope, "project", "topic", ppMessages)
 When I try to publish a message in a topic I get an error message
 "Could not find the sink for pubsub, Check that the sink library
 specifies alwayslink = 1

 I found a StackOverFlow post for the same issue but it doesn't
 solve the problem.

 Stackoverflow Link
 

 Can someone please help?

 Regards,
 Ashok

>>>

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

[Ashok KS]

Yes, that is where Iam getting stuck. I wrote the complete pipeline in
Python which reads from the BQ table and published it as a PubSub message.
I'm able to force it as a streaming application by passing --streaming=True
But for my project, they want it in Go so I had to rewrite the complete
logic in Go.
I did the same, but stuck at the last point of publishing it to PubSub.

On Mon, Feb 6, 2023 at 11:07 PM Shivam Singhal 
wrote:

> It depends on the input source: it will decide if your pipeline is a
> streaming or a batch pipeline.
>
> Since you are querying over a BQ table, the input is finite and in result,
> your pipeline is a batch pipeline.
> I am not sure there is a straightforward way where you can convert this
> pipeline into a streaming pipeline.
>
>
> On Mon, 6 Feb 2023 at 17:32, Ashok KS  wrote:
>
>> Hi Shivam,
>>
>> Thanks for that. How can run the pipeline as a streaming pipeline?  In
>> python I could just run the pipeline by passing —streaming=True in the
>> command line, but I couldn’t find anything similar in Go.
>>
>> Any pointers would be appreciated.
>>
>> Regards,
>> Ashok
>>
>> On Mon, 6 Feb 2023 at 10:59 pm, Shivam Singhal <
>> shivamsinghal5...@gmail.com> wrote:
>>
>>> The issue is not yet verified by the maintainers but I think the
>>> pubsubio connector's Write method doesn't work in Batch pipelines.
>>>
>>> But I am pretty sure that pubsubio Write doesn't work for Batch
>>> Pipelines because it's mentioned in the code comments. Check the below
>>> issue for the details:
>>> https://github.com/apache/beam/issues/25326
>>>
>>> On Mon, 6 Feb 2023 at 17:26, Ashok KS  wrote:
>>>
 Hi Shivam,

 Thanks a lot for your response. Yes it is a batch pipeline. My task is
 to read a big query table, process the data and publish the Rows as a
 PubSub message.

 Regards,
 Ashok

 On Mon, 6 Feb 2023 at 10:52 pm, Shivam Singhal <
 shivamsinghal5...@gmail.com> wrote:

> Hey Ashok KS,
>
> Is this a batch pipeline?
>
> On Mon, 6 Feb 2023 at 09:27, Ashok KS  wrote:
>
>> Hi All,
>>
>> Just sending a reminder in case anyone could help. I haven't received
>> any response to my issue.
>>
>> Regards,
>> Ashok
>>
>> On Fri, Feb 3, 2023 at 12:23 AM Ashok KS  wrote:
>>
>>> Hi All,
>>>
>>> I'm new to using Apache Beam using Go.
>>>
>>> pubsubio.Write(scope, "project", "topic", ppMessages)
>>> When I try to publish a message in a topic I get an error message
>>> "Could not find the sink for pubsub, Check that the sink library
>>> specifies alwayslink = 1
>>>
>>> I found a StackOverFlow post for the same issue but it doesn't solve
>>> the problem.
>>>
>>> Stackoverflow Link
>>> 
>>>
>>> Can someone please help?
>>>
>>> Regards,
>>> Ashok
>>>
>>

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

[Shivam Singhal]

It depends on the input source: it will decide if your pipeline is a
streaming or a batch pipeline.

Since you are querying over a BQ table, the input is finite and in result,
your pipeline is a batch pipeline.
I am not sure there is a straightforward way where you can convert this
pipeline into a streaming pipeline.


On Mon, 6 Feb 2023 at 17:32, Ashok KS  wrote:

> Hi Shivam,
>
> Thanks for that. How can run the pipeline as a streaming pipeline?  In
> python I could just run the pipeline by passing —streaming=True in the
> command line, but I couldn’t find anything similar in Go.
>
> Any pointers would be appreciated.
>
> Regards,
> Ashok
>
> On Mon, 6 Feb 2023 at 10:59 pm, Shivam Singhal <
> shivamsinghal5...@gmail.com> wrote:
>
>> The issue is not yet verified by the maintainers but I think the pubsubio
>> connector's Write method doesn't work in Batch pipelines.
>>
>> But I am pretty sure that pubsubio Write doesn't work for Batch Pipelines
>> because it's mentioned in the code comments. Check the below issue for the
>> details:
>> https://github.com/apache/beam/issues/25326
>>
>> On Mon, 6 Feb 2023 at 17:26, Ashok KS  wrote:
>>
>>> Hi Shivam,
>>>
>>> Thanks a lot for your response. Yes it is a batch pipeline. My task is
>>> to read a big query table, process the data and publish the Rows as a
>>> PubSub message.
>>>
>>> Regards,
>>> Ashok
>>>
>>> On Mon, 6 Feb 2023 at 10:52 pm, Shivam Singhal <
>>> shivamsinghal5...@gmail.com> wrote:
>>>
 Hey Ashok KS,

 Is this a batch pipeline?

 On Mon, 6 Feb 2023 at 09:27, Ashok KS  wrote:

> Hi All,
>
> Just sending a reminder in case anyone could help. I haven't received
> any response to my issue.
>
> Regards,
> Ashok
>
> On Fri, Feb 3, 2023 at 12:23 AM Ashok KS  wrote:
>
>> Hi All,
>>
>> I'm new to using Apache Beam using Go.
>>
>> pubsubio.Write(scope, "project", "topic", ppMessages)
>> When I try to publish a message in a topic I get an error message
>> "Could not find the sink for pubsub, Check that the sink library
>> specifies alwayslink = 1
>>
>> I found a StackOverFlow post for the same issue but it doesn't solve
>> the problem.
>>
>> Stackoverflow Link
>> 
>>
>> Can someone please help?
>>
>> Regards,
>> Ashok
>>
>

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

[Ashok KS]

Hi Shivam,

Thanks for that. How can run the pipeline as a streaming pipeline?  In
python I could just run the pipeline by passing —streaming=True in the
command line, but I couldn’t find anything similar in Go.

Any pointers would be appreciated.

Regards,
Ashok

On Mon, 6 Feb 2023 at 10:59 pm, Shivam Singhal 
wrote:

> The issue is not yet verified by the maintainers but I think the pubsubio
> connector's Write method doesn't work in Batch pipelines.
>
> But I am pretty sure that pubsubio Write doesn't work for Batch Pipelines
> because it's mentioned in the code comments. Check the below issue for the
> details:
> https://github.com/apache/beam/issues/25326
>
> On Mon, 6 Feb 2023 at 17:26, Ashok KS  wrote:
>
>> Hi Shivam,
>>
>> Thanks a lot for your response. Yes it is a batch pipeline. My task is to
>> read a big query table, process the data and publish the Rows as a PubSub
>> message.
>>
>> Regards,
>> Ashok
>>
>> On Mon, 6 Feb 2023 at 10:52 pm, Shivam Singhal <
>> shivamsinghal5...@gmail.com> wrote:
>>
>>> Hey Ashok KS,
>>>
>>> Is this a batch pipeline?
>>>
>>> On Mon, 6 Feb 2023 at 09:27, Ashok KS  wrote:
>>>
 Hi All,

 Just sending a reminder in case anyone could help. I haven't received
 any response to my issue.

 Regards,
 Ashok

 On Fri, Feb 3, 2023 at 12:23 AM Ashok KS  wrote:

> Hi All,
>
> I'm new to using Apache Beam using Go.
>
> pubsubio.Write(scope, "project", "topic", ppMessages)
> When I try to publish a message in a topic I get an error message
> "Could not find the sink for pubsub, Check that the sink library
> specifies alwayslink = 1
>
> I found a StackOverFlow post for the same issue but it doesn't solve
> the problem.
>
> Stackoverflow Link
> 
>
> Can someone please help?
>
> Regards,
> Ashok
>

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

[Shivam Singhal]

The issue is not yet verified by the maintainers but I think the pubsubio
connector's Write method doesn't work in Batch pipelines.

But I am pretty sure that pubsubio Write doesn't work for Batch Pipelines
because it's mentioned in the code comments. Check the below issue for the
details:
https://github.com/apache/beam/issues/25326

On Mon, 6 Feb 2023 at 17:26, Ashok KS  wrote:

> Hi Shivam,
>
> Thanks a lot for your response. Yes it is a batch pipeline. My task is to
> read a big query table, process the data and publish the Rows as a PubSub
> message.
>
> Regards,
> Ashok
>
> On Mon, 6 Feb 2023 at 10:52 pm, Shivam Singhal <
> shivamsinghal5...@gmail.com> wrote:
>
>> Hey Ashok KS,
>>
>> Is this a batch pipeline?
>>
>> On Mon, 6 Feb 2023 at 09:27, Ashok KS  wrote:
>>
>>> Hi All,
>>>
>>> Just sending a reminder in case anyone could help. I haven't received
>>> any response to my issue.
>>>
>>> Regards,
>>> Ashok
>>>
>>> On Fri, Feb 3, 2023 at 12:23 AM Ashok KS  wrote:
>>>
 Hi All,

 I'm new to using Apache Beam using Go.

 pubsubio.Write(scope, "project", "topic", ppMessages)
 When I try to publish a message in a topic I get an error message
 "Could not find the sink for pubsub, Check that the sink library
 specifies alwayslink = 1

 I found a StackOverFlow post for the same issue but it doesn't solve
 the problem.

 Stackoverflow Link
 

 Can someone please help?

 Regards,
 Ashok

>>>

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

[Ashok KS]

Hi Shivam,

Thanks a lot for your response. Yes it is a batch pipeline. My task is to
read a big query table, process the data and publish the Rows as a PubSub
message.

Regards,
Ashok

On Mon, 6 Feb 2023 at 10:52 pm, Shivam Singhal 
wrote:

> Hey Ashok KS,
>
> Is this a batch pipeline?
>
> On Mon, 6 Feb 2023 at 09:27, Ashok KS  wrote:
>
>> Hi All,
>>
>> Just sending a reminder in case anyone could help. I haven't received any
>> response to my issue.
>>
>> Regards,
>> Ashok
>>
>> On Fri, Feb 3, 2023 at 12:23 AM Ashok KS  wrote:
>>
>>> Hi All,
>>>
>>> I'm new to using Apache Beam using Go.
>>>
>>> pubsubio.Write(scope, "project", "topic", ppMessages)
>>> When I try to publish a message in a topic I get an error message
>>> "Could not find the sink for pubsub, Check that the sink library
>>> specifies alwayslink = 1
>>>
>>> I found a StackOverFlow post for the same issue but it doesn't solve the
>>> problem.
>>>
>>> Stackoverflow Link
>>> 
>>>
>>> Can someone please help?
>>>
>>> Regards,
>>> Ashok
>>>
>>

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

[Shivam Singhal]

Hey Ashok KS,

Is this a batch pipeline?

On Mon, 6 Feb 2023 at 09:27, Ashok KS  wrote:

> Hi All,
>
> Just sending a reminder in case anyone could help. I haven't received any
> response to my issue.
>
> Regards,
> Ashok
>
> On Fri, Feb 3, 2023 at 12:23 AM Ashok KS  wrote:
>
>> Hi All,
>>
>> I'm new to using Apache Beam using Go.
>>
>> pubsubio.Write(scope, "project", "topic", ppMessages)
>> When I try to publish a message in a topic I get an error message
>> "Could not find the sink for pubsub, Check that the sink library
>> specifies alwayslink = 1
>>
>> I found a StackOverFlow post for the same issue but it doesn't solve the
>> problem.
>>
>> Stackoverflow Link
>> 
>>
>> Can someone please help?
>>
>> Regards,
>> Ashok
>>
>

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

[Ashok KS]

Hi All,

Just sending a reminder in case anyone could help. I haven't received any
response to my issue.

Regards,
Ashok

On Fri, Feb 3, 2023 at 12:23 AM Ashok KS  wrote:

> Hi All,
>
> I'm new to using Apache Beam using Go.
>
> pubsubio.Write(scope, "project", "topic", ppMessages)
> When I try to publish a message in a topic I get an error message
> "Could not find the sink for pubsub, Check that the sink library specifies
> alwayslink = 1
>
> I found a StackOverFlow post for the same issue but it doesn't solve the
> problem.
>
> Stackoverflow Link
> 
>
> Can someone please help?
>
> Regards,
> Ashok
>

Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

[Ashok KS]

Hi All,

I'm new to using Apache Beam using Go.

pubsubio.Write(scope, "project", "topic", ppMessages)
When I try to publish a message in a topic I get an error message
"Could not find the sink for pubsub, Check that the sink library specifies
alwayslink = 1

I found a StackOverFlow post for the same issue but it doesn't solve the
problem.

Stackoverflow Link


Can someone please help?

Regards,
Ashok