Re: Bigquery Connector Rate limits
Hello Ahmed, Thanks for the information this helps a lot. On Thu, 22 Feb 2024 at 9:09 PM, Ahmed Abualsaud via dev wrote: > Hey Taher, > > Regarding the first question about what API Beam uses, that depends on the > BigQuery method you set in the connector's configuration. We have 4 > different write methods, and a high-level description of each can be found > in the documentation: > https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.Method.html. > At this point in time, we discourage using the streaming inserts API and > recommend file loads or Storage Write API instead. > > For the second question, yes there is a chance you can hit the maximum > quota. When this happens, Beam will just wait a little then retry the write > operation. FYI the Storage Write API quota [1] limits to 3gb/s per project, > compared to streaming insert's 1gb/s [2]. > > [1] https://cloud.google.com/bigquery/quotas#write-api-limits > [2] https://cloud.google.com/bigquery/quotas#streaming_inserts > > On Thu, Feb 22, 2024 at 8:57 AM Taher Koitawala > wrote: > >> Hi All, >> I want to ask questions regarding sinking a very high volume >> stream to Bigquery. >> >> I will read messages from a Pubsub topic and write to Bigquery. In this >> steaming job i am worried about hitting the bigquery streaming inserts >> limit of 1gb per second on streaming Api writes >> >> I am firstly unsure if Beam uses that Api or uses a temp directory to >> write files and commits on intervals which brings me to another question do >> i have to do windowing to save myself from hitting the 1gb per second >> limit? >> >> Please advise. Thanks >> >
Bigquery Connector Rate limits
Hi All, I want to ask questions regarding sinking a very high volume stream to Bigquery. I will read messages from a Pubsub topic and write to Bigquery. In this steaming job i am worried about hitting the bigquery streaming inserts limit of 1gb per second on streaming Api writes I am firstly unsure if Beam uses that Api or uses a temp directory to write files and commits on intervals which brings me to another question do i have to do windowing to save myself from hitting the 1gb per second limit? Please advise. Thanks
Re: [DISCUSS] Query external resources as Tables with Beam SQL
Also auto creation is not there
On Thu, Mar 5, 2020 at 3:59 PM Taher Koitawala wrote:
> Proposal is to add more sources and also have time event time or
> processing enhancements further on them
>
> On Thu, Mar 5, 2020 at 3:50 PM Andrew Pilloud wrote:
>
>> I believe we have this functionality alredy:
>> https://beam.apache.org/documentation/dsls/sql/extensions/create-external-table/
>>
>> Existing GCP tables can also be loaded through the GCP datacatalog
>> metastore. What are you proposing that is new?
>>
>> Andrew
>>
>>
>> On Thu, Mar 5, 2020, 12:29 AM Taher Koitawala wrote:
>>
>>> Hi All,
>>> We have been using Apache Beam extensively to process huge
>>> amounts of data, while beam is really powerful and can solve a huge number
>>> of use cases. A Beam job's development and testing time is significantly
>>> high.
>>>
>>>This gap can be filled with Beam SQL, where a complete SQL based
>>> interface can reduce development and testing time to matter of minutes, it
>>> also makes Apache Beam more user friendly where a wide variety of audience
>>> with different analytical skillsets can interact.
>>>
>>> The current Beam SQL is still needs to be used programmatically, and so
>>> I propose the following additions/improvements.
>>>
>>> *Note: Whist the below given examples are more GCP biased, they apply to
>>> other sources in a generic manner*
>>>
>>> For Example: Imagine a user who wants to write a stream processing job
>>> on Google Cloud Dataflow. The user wants to process credit card transaction
>>> streams from Google Cloud PubSub (Something like Kafka) and enrich each
>>> record of the stream with some data that is stored in Google Cloud Spanner,
>>> after enrichment the user wishes to write the following data to Google
>>> Cloud BigQuery.
>>>
>>> Given Below are the queries which the user should be able to fire on
>>> Beam and the rest should be automatically handled by the framework.
>>>
>>> //Infer schema from Spanner table upon table creation
>>>
>>> CREATE TABLE SPANNER_CARD_INFO
>>>
>>> OPTIONS (
>>>
>>> ProjectId: “gcp-project”,
>>>
>>> InstanceId : “spanner-instance-id”,
>>>
>>> Database: “some-database”,
>>>
>>> Table: “card_info”,
>>>
>>> CloudResource: “SPANNER”,
>>>
>>> CreateTableIfNotExists: “FALSE”
>>>
>>> )
>>> //Apply schema to each record read from pubsub, and then apply SQL.
>>>
>>> CREATE TABLE TRANSACTIONS_PUBSUB_TOPIC
>>>
>>> OPTIONS (
>>>
>>> ProjectId: “gcp-project”,
>>>
>>> Topic: “card-transactions”,
>>>
>>> CloudResource : “PUBSUB”
>>>
>>> SubscriptionId : “subscriptionId-1”,
>>>
>>> CreateTopicIfNotExists: “FALSE”,
>>>
>>> CreateSubscriptionIfNotExist: “TRUE”,
>>>
>>> RecordType: “JSON” //POssible values: Avro, JSON, TVS..etc
>>>
>>> JsonRecordSchema : “{
>>>
>>> “CardNumber” : “INT”,
>>>
>>> “Amount”: “DOUBLE”,
>>>
>>> “eventTimeStamp” : “EVENT_TIME”
>>>
>>> }”)
>>>
>>> //Create table in BigQuery if not exists and insert
>>>
>>> CREATE TABLE TRANSACTION_HISTORY
>>>
>>> OPTIONS (
>>>
>>> ProjectId: “gcp-project”,
>>>
>>> CloudResource : “BIGQUERY”
>>>
>>> dataset: “dataset1”,
>>>
>>> table : “table1”,
>>>
>>> CreateTableIfNotExists: “TRUE”,
>>>
>>> TableSchema : “
>>>
>>> {
>>>
>>> “card_number” : “INT”,
>>>
>>> “first_name” : “STRING”,
>>>
>>> “last_name” : “STRING”,
>>>
>>> “phone” : “INT”,
>>>
>>> “city” : “STRING”,
>>>
>>> “amount”: “FLOAT”,
>>>
>>> “eventtimestamp” : “INT”,
>>>
>>> }”)
>>>
>>> //Actual query that should get stretched to a Beam dag
>>>
>>> INSERT INTO TRANSACTION_HISTORY
>>>
>>> SELECT
>>> pubsub.card_number,spanner.first_name,spanner.last_name,spanner.phone,spanner.city,pubsub.amount,pubsub.eventTimeStamp
>>> FROM TRANSACTIONS_PUBSUB_TOPIC pubsub join SPANNER_CARD_INFO spanner on
>>> (pubsub.card_number = spanner.card_number);
>>>
>>>
>>>
>>> Also to consider that if any of the sources or sinks change, we only
>>> change the SQL and done!.
>>>
>>> Please let me know your thoughts about this.
>>>
>>> Regards,
>>> Taher Koitawala
>>>
>>>
Re: [DISCUSS] Query external resources as Tables with Beam SQL
Proposal is to add more sources and also have time event time or processing
enhancements further on them
On Thu, Mar 5, 2020 at 3:50 PM Andrew Pilloud wrote:
> I believe we have this functionality alredy:
> https://beam.apache.org/documentation/dsls/sql/extensions/create-external-table/
>
> Existing GCP tables can also be loaded through the GCP datacatalog
> metastore. What are you proposing that is new?
>
> Andrew
>
>
> On Thu, Mar 5, 2020, 12:29 AM Taher Koitawala wrote:
>
>> Hi All,
>> We have been using Apache Beam extensively to process huge
>> amounts of data, while beam is really powerful and can solve a huge number
>> of use cases. A Beam job's development and testing time is significantly
>> high.
>>
>>This gap can be filled with Beam SQL, where a complete SQL based
>> interface can reduce development and testing time to matter of minutes, it
>> also makes Apache Beam more user friendly where a wide variety of audience
>> with different analytical skillsets can interact.
>>
>> The current Beam SQL is still needs to be used programmatically, and so I
>> propose the following additions/improvements.
>>
>> *Note: Whist the below given examples are more GCP biased, they apply to
>> other sources in a generic manner*
>>
>> For Example: Imagine a user who wants to write a stream processing job on
>> Google Cloud Dataflow. The user wants to process credit card transaction
>> streams from Google Cloud PubSub (Something like Kafka) and enrich each
>> record of the stream with some data that is stored in Google Cloud Spanner,
>> after enrichment the user wishes to write the following data to Google
>> Cloud BigQuery.
>>
>> Given Below are the queries which the user should be able to fire on Beam
>> and the rest should be automatically handled by the framework.
>>
>> //Infer schema from Spanner table upon table creation
>>
>> CREATE TABLE SPANNER_CARD_INFO
>>
>> OPTIONS (
>>
>> ProjectId: “gcp-project”,
>>
>> InstanceId : “spanner-instance-id”,
>>
>> Database: “some-database”,
>>
>> Table: “card_info”,
>>
>> CloudResource: “SPANNER”,
>>
>> CreateTableIfNotExists: “FALSE”
>>
>> )
>> //Apply schema to each record read from pubsub, and then apply SQL.
>>
>> CREATE TABLE TRANSACTIONS_PUBSUB_TOPIC
>>
>> OPTIONS (
>>
>> ProjectId: “gcp-project”,
>>
>> Topic: “card-transactions”,
>>
>> CloudResource : “PUBSUB”
>>
>> SubscriptionId : “subscriptionId-1”,
>>
>> CreateTopicIfNotExists: “FALSE”,
>>
>> CreateSubscriptionIfNotExist: “TRUE”,
>>
>> RecordType: “JSON” //POssible values: Avro, JSON, TVS..etc
>>
>> JsonRecordSchema : “{
>>
>> “CardNumber” : “INT”,
>>
>> “Amount”: “DOUBLE”,
>>
>> “eventTimeStamp” : “EVENT_TIME”
>>
>> }”)
>>
>> //Create table in BigQuery if not exists and insert
>>
>> CREATE TABLE TRANSACTION_HISTORY
>>
>> OPTIONS (
>>
>> ProjectId: “gcp-project”,
>>
>> CloudResource : “BIGQUERY”
>>
>> dataset: “dataset1”,
>>
>> table : “table1”,
>>
>> CreateTableIfNotExists: “TRUE”,
>>
>> TableSchema : “
>>
>> {
>>
>> “card_number” : “INT”,
>>
>> “first_name” : “STRING”,
>>
>> “last_name” : “STRING”,
>>
>> “phone” : “INT”,
>>
>> “city” : “STRING”,
>>
>> “amount”: “FLOAT”,
>>
>> “eventtimestamp” : “INT”,
>>
>> }”)
>>
>> //Actual query that should get stretched to a Beam dag
>>
>> INSERT INTO TRANSACTION_HISTORY
>>
>> SELECT
>> pubsub.card_number,spanner.first_name,spanner.last_name,spanner.phone,spanner.city,pubsub.amount,pubsub.eventTimeStamp
>> FROM TRANSACTIONS_PUBSUB_TOPIC pubsub join SPANNER_CARD_INFO spanner on
>> (pubsub.card_number = spanner.card_number);
>>
>>
>>
>> Also to consider that if any of the sources or sinks change, we only
>> change the SQL and done!.
>>
>> Please let me know your thoughts about this.
>>
>> Regards,
>> Taher Koitawala
>>
>>
[DISCUSS] Query external resources as Tables with Beam SQL
Hi All,
We have been using Apache Beam extensively to process huge amounts
of data, while beam is really powerful and can solve a huge number of use
cases. A Beam job's development and testing time is significantly high.
This gap can be filled with Beam SQL, where a complete SQL based
interface can reduce development and testing time to matter of minutes, it
also makes Apache Beam more user friendly where a wide variety of audience
with different analytical skillsets can interact.
The current Beam SQL is still needs to be used programmatically, and so I
propose the following additions/improvements.
*Note: Whist the below given examples are more GCP biased, they apply to
other sources in a generic manner*
For Example: Imagine a user who wants to write a stream processing job on
Google Cloud Dataflow. The user wants to process credit card transaction
streams from Google Cloud PubSub (Something like Kafka) and enrich each
record of the stream with some data that is stored in Google Cloud Spanner,
after enrichment the user wishes to write the following data to Google
Cloud BigQuery.
Given Below are the queries which the user should be able to fire on Beam
and the rest should be automatically handled by the framework.
//Infer schema from Spanner table upon table creation
CREATE TABLE SPANNER_CARD_INFO
OPTIONS (
ProjectId: “gcp-project”,
InstanceId : “spanner-instance-id”,
Database: “some-database”,
Table: “card_info”,
CloudResource: “SPANNER”,
CreateTableIfNotExists: “FALSE”
)
//Apply schema to each record read from pubsub, and then apply SQL.
CREATE TABLE TRANSACTIONS_PUBSUB_TOPIC
OPTIONS (
ProjectId: “gcp-project”,
Topic: “card-transactions”,
CloudResource : “PUBSUB”
SubscriptionId : “subscriptionId-1”,
CreateTopicIfNotExists: “FALSE”,
CreateSubscriptionIfNotExist: “TRUE”,
RecordType: “JSON” //POssible values: Avro, JSON, TVS..etc
JsonRecordSchema : “{
“CardNumber” : “INT”,
“Amount”: “DOUBLE”,
“eventTimeStamp” : “EVENT_TIME”
}”)
//Create table in BigQuery if not exists and insert
CREATE TABLE TRANSACTION_HISTORY
OPTIONS (
ProjectId: “gcp-project”,
CloudResource : “BIGQUERY”
dataset: “dataset1”,
table : “table1”,
CreateTableIfNotExists: “TRUE”,
TableSchema : “
{
“card_number” : “INT”,
“first_name” : “STRING”,
“last_name” : “STRING”,
“phone” : “INT”,
“city” : “STRING”,
“amount”: “FLOAT”,
“eventtimestamp” : “INT”,
}”)
//Actual query that should get stretched to a Beam dag
INSERT INTO TRANSACTION_HISTORY
SELECT
pubsub.card_number,spanner.first_name,spanner.last_name,spanner.phone,spanner.city,pubsub.amount,pubsub.eventTimeStamp
FROM TRANSACTIONS_PUBSUB_TOPIC pubsub join SPANNER_CARD_INFO spanner on
(pubsub.card_number = spanner.card_number);
Also to consider that if any of the sources or sinks change, we only change
the SQL and done!.
Please let me know your thoughts about this.
Regards,
Taher Koitawala
Streaming data from Pubsub to Spanner with Beam dataflow pipeline
Hi All, My current use-case is to write data from Pubsub to Spanner using a streaming pipeline. I do see that Beam does have a SpannerIO to write. However, pubsub being streaming and Spanner being RDBMS like, it would be helpful to you guys can tell me if this will be performant enough or not. If someone has already tried this out and can give me a few caveats, then that would be really awesome. Regards, Taher Koitawala
Re: Apache Pulsar connector for Beam
Awesome. Thanks Max. Will definitely keep the community posted on this. On Sat, Oct 26, 2019, 4:21 PM Maximilian Michels wrote: > Awesome. I've made you a contributor, so you can also create issues and > assign yourself now. > > Please let us know about the progress once you start writing the connector. > > Thanks, > Max > > On 26.10.19 12:36, Taher Koitawala wrote: > > Thank you Alex and Max, > > My jira id is taherk77. Please add me. > > > > Regards, > > Taher Koitawala > > > > On Sat, Oct 26, 2019, 3:53 PM Maximilian Michels > <mailto:m...@apache.org>> wrote: > > > > That sounds great. How about you start looking into this Taher? If > > necessary, Sijie could provide additional insight into Pulsar. > > > > Please create a JIRA account so we can assign you to > > https://issues.apache.org/jira/browse/BEAM-8218 > > > > Thanks, > > Max > > > > On 26.10.19 12:08, Alex Van Boxel wrote: > > > Hey Taher, do you have a Jira account? Then I will assign the > > ticket to > > > you. I made the ticket because we should have one, feel free to > > take the > > > lead on this one. > > > > > > _/ > > > _/ Alex Van Boxel > > > > > > > > > On Fri, Oct 25, 2019 at 9:35 PM Taher Koitawala > > mailto:taher...@gmail.com> > > > <mailto:taher...@gmail.com <mailto:taher...@gmail.com>>> wrote: > > > > > > I would be interested in contributing to the Pulsar Beam > > connector. > > > That's one of the reasons i started the email thread. > > > > > > > > > Regards, > > > Taher Koitawala > > > > > > On Sat, Oct 26, 2019, 9:41 AM Sijie Guo > <mailto:s...@streamnative.io> > > > <mailto:s...@streamnative.io <mailto:s...@streamnative.io>>> > wrote: > > > > > > This is Sijie Guo from StreamNative and Pulsar PMC. > > > > > > Maximilian - thank you for adding us in the email thread! > > > > > > We do have one roadmap item for adding a Beam connector > for > > > Pulsar. It was planned for this quarter, but we haven’t > > started > > > the implementation yet. If the Beam community is > > interested in > > > it, we are happy to collaborate with Beam community. > > > > > > Thanks, > > > Sijie > > > > > > On Sat, Oct 26, 2019 at 12:36 AM Maximilian Michels > > > mailto:m...@apache.org> > > <mailto:m...@apache.org <mailto:m...@apache.org>>> wrote: > > > > > > It would be great to have a Pulsar connector. We > > might want > > > to ask the > > > folks from StreamNative (in CC). Any plans? :) > > > > > > Cheers, > > > Max > > > > > > On 24.10.19 18:31, Pablo Estrada wrote: > > > > There's a JIRA issue to track this: > > > > https://issues.apache.org/jira/browse/BEAM-8218 > > > > > > > > Alex was kind enough to file it. +Alex Van Boxel > > > > <mailto:a...@vanboxel.be <mailto:a...@vanboxel.be> > > <mailto:a...@vanboxel.be <mailto:a...@vanboxel.be>>> : ) > > > > Best > > > > -P > > > > > > > > On Thu, Oct 24, 2019 at 12:01 AM Taher Koitawala > > > mailto:taher...@gmail.com> > > <mailto:taher...@gmail.com <mailto:taher...@gmail.com>> > > > > <mailto:taher...@gmail.com > > <mailto:taher...@gmail.com> <mailto:taher...@gmail.com > > <mailto:taher...@gmail.com>>>> > > > wrote: > > > > > > > > Hi Reza, > > > > Thanks for your reply. However i > > do not > > > see Pulsar > > > > listed in there. Should we
Re: Apache Pulsar connector for Beam
Thank you Alex and Max, My jira id is taherk77. Please add me. Regards, Taher Koitawala On Sat, Oct 26, 2019, 3:53 PM Maximilian Michels wrote: > That sounds great. How about you start looking into this Taher? If > necessary, Sijie could provide additional insight into Pulsar. > > Please create a JIRA account so we can assign you to > https://issues.apache.org/jira/browse/BEAM-8218 > > Thanks, > Max > > On 26.10.19 12:08, Alex Van Boxel wrote: > > Hey Taher, do you have a Jira account? Then I will assign the ticket to > > you. I made the ticket because we should have one, feel free to take the > > lead on this one. > > > > _/ > > _/ Alex Van Boxel > > > > > > On Fri, Oct 25, 2019 at 9:35 PM Taher Koitawala > <mailto:taher...@gmail.com>> wrote: > > > > I would be interested in contributing to the Pulsar Beam connector. > > That's one of the reasons i started the email thread. > > > > > > Regards, > > Taher Koitawala > > > > On Sat, Oct 26, 2019, 9:41 AM Sijie Guo > <mailto:s...@streamnative.io>> wrote: > > > > This is Sijie Guo from StreamNative and Pulsar PMC. > > > > Maximilian - thank you for adding us in the email thread! > > > > We do have one roadmap item for adding a Beam connector for > > Pulsar. It was planned for this quarter, but we haven’t started > > the implementation yet. If the Beam community is interested in > > it, we are happy to collaborate with Beam community. > > > > Thanks, > > Sijie > > > > On Sat, Oct 26, 2019 at 12:36 AM Maximilian Michels > > mailto:m...@apache.org>> wrote: > > > > It would be great to have a Pulsar connector. We might want > > to ask the > > folks from StreamNative (in CC). Any plans? :) > > > > Cheers, > > Max > > > > On 24.10.19 18:31, Pablo Estrada wrote: > > > There's a JIRA issue to track this: > > > https://issues.apache.org/jira/browse/BEAM-8218 > > > > > > Alex was kind enough to file it. +Alex Van Boxel > > > <mailto:a...@vanboxel.be <mailto:a...@vanboxel.be>> : ) > > > Best > > > -P > > > > > > On Thu, Oct 24, 2019 at 12:01 AM Taher Koitawala > > mailto:taher...@gmail.com> > > > <mailto:taher...@gmail.com <mailto:taher...@gmail.com>>> > > wrote: > > > > > > Hi Reza, > > > Thanks for your reply. However i do not > > see Pulsar > > > listed in there. Should we file a jira? > > > > > > On Thu, Oct 24, 2019, 12:16 PM Reza Rokni > > mailto:r...@google.com> > > > <mailto:r...@google.com <mailto:r...@google.com>>> > wrote: > > > > > > Hi Taher, > > > > > > You can see the list of current and wip IO's here: > > > > > > https://beam.apache.org/documentation/io/built-in/ > > > > > > Cheers > > > > > > Reza > > > > > > On Thu, 24 Oct 2019 at 13:56, Taher Koitawala > > > mailto:taher...@gmail.com> > > <mailto:taher...@gmail.com <mailto:taher...@gmail.com>>> > wrote: > > > > > > Hi All, > > > Been wanting to know if we have a > > Pulsar connector > > > for Beam. Pulsar is another messaging queue > > like Kafka and I > > > would like to build a streaming pipeline with > > Pulsar. Any > > > help would be appreciated.. > > > > > > > > > Regards, > > > Taher Koitawala > > > > > > > > > > > > -- > >
Re: Apache Pulsar connector for Beam
I would be interested in contributing to the Pulsar Beam connector. That's one of the reasons i started the email thread. Regards, Taher Koitawala On Sat, Oct 26, 2019, 9:41 AM Sijie Guo wrote: > This is Sijie Guo from StreamNative and Pulsar PMC. > > Maximilian - thank you for adding us in the email thread! > > We do have one roadmap item for adding a Beam connector for Pulsar. It was > planned for this quarter, but we haven’t started the implementation yet. If > the Beam community is interested in it, we are happy to collaborate with > Beam community. > > Thanks, > Sijie > > On Sat, Oct 26, 2019 at 12:36 AM Maximilian Michels > wrote: > >> It would be great to have a Pulsar connector. We might want to ask the >> folks from StreamNative (in CC). Any plans? :) >> >> Cheers, >> Max >> >> On 24.10.19 18:31, Pablo Estrada wrote: >> > There's a JIRA issue to track this: >> > https://issues.apache.org/jira/browse/BEAM-8218 >> > >> > Alex was kind enough to file it. +Alex Van Boxel >> > <mailto:a...@vanboxel.be> : ) >> > Best >> > -P >> > >> > On Thu, Oct 24, 2019 at 12:01 AM Taher Koitawala > > <mailto:taher...@gmail.com>> wrote: >> > >> > Hi Reza, >> > Thanks for your reply. However i do not see Pulsar >> > listed in there. Should we file a jira? >> > >> > On Thu, Oct 24, 2019, 12:16 PM Reza Rokni > > <mailto:r...@google.com>> wrote: >> > >> > Hi Taher, >> > >> > You can see the list of current and wip IO's here: >> > >> > https://beam.apache.org/documentation/io/built-in/ >> > >> > Cheers >> > >> > Reza >> > >> > On Thu, 24 Oct 2019 at 13:56, Taher Koitawala >> > mailto:taher...@gmail.com>> wrote: >> > >> > Hi All, >> > Been wanting to know if we have a Pulsar connector >> > for Beam. Pulsar is another messaging queue like Kafka and I >> > would like to build a streaming pipeline with Pulsar. Any >> > help would be appreciated.. >> > >> > >> > Regards, >> > Taher Koitawala >> > >> > >> > >> > -- >> > >> > This email may be confidential and privileged. If you received >> > this communication by mistake, please don't forward it to anyone >> > else, please erase all copies and attachments, and please let me >> > know that it has gone to the wrong person. >> > >> > The above terms reflect a potential business arrangement, are >> > provided solely as a basis for further discussion, and are not >> > intended to be and do not constitute a legally binding >> > obligation. No legally binding obligations will be created, >> > implied, or inferred until an agreement in final form is >> > executed in writing by all parties involved. >> > >> >
Re: Apache Pulsar connector for Beam
Hi Reza, Thanks for your reply. However i do not see Pulsar listed in there. Should we file a jira? On Thu, Oct 24, 2019, 12:16 PM Reza Rokni wrote: > Hi Taher, > > You can see the list of current and wip IO's here: > > https://beam.apache.org/documentation/io/built-in/ > > Cheers > > Reza > > On Thu, 24 Oct 2019 at 13:56, Taher Koitawala wrote: > >> Hi All, >> Been wanting to know if we have a Pulsar connector for Beam. >> Pulsar is another messaging queue like Kafka and I would like to build a >> streaming pipeline with Pulsar. Any help would be appreciated.. >> >> >> Regards, >> Taher Koitawala >> > > > -- > > This email may be confidential and privileged. If you received this > communication by mistake, please don't forward it to anyone else, please > erase all copies and attachments, and please let me know that it has gone > to the wrong person. > > The above terms reflect a potential business arrangement, are provided > solely as a basis for further discussion, and are not intended to be and do > not constitute a legally binding obligation. No legally binding obligations > will be created, implied, or inferred until an agreement in final form is > executed in writing by all parties involved. >
Apache Pulsar connector for Beam
Hi All, Been wanting to know if we have a Pulsar connector for Beam. Pulsar is another messaging queue like Kafka and I would like to build a streaming pipeline with Pulsar. Any help would be appreciated.. Regards, Taher Koitawala
