[jira] [Commented] (BEAM-5676) Allow to change the PubsubClientFactory when using PubsubIO
[
https://issues.apache.org/jira/browse/BEAM-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739597#comment-16739597
]
Logan HAUSPIE commented on BEAM-5676:
-
I will do it next month if no one else has done it before me
> Allow to change the PubsubClientFactory when using PubsubIO
> ---
>
> Key: BEAM-5676
> URL: https://issues.apache.org/jira/browse/BEAM-5676
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
>Affects Versions: 2.7.0
>Reporter: Logan HAUSPIE
>Priority: Major
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> When we use PubSub to push or pull messages, we currently have no choice of
> serialization implementation because PubsubIO use internaly the
> *_PubsubJsonClient.FACTORY_* of type _*PubsubJsonClientFactory*_ to serialize
> or deserialize messages.
> It should be nice to be able to set a different factory (e.g.
> *_PubsubGrpcClientFactory_*) to decrease the size of messages (and then
> decrease the price of using Pubsub).
>
> I guess It could be possible to write something like:
> {{PubsubIO.Write write =}}
> {{ PubsubIO.writeMessages()}}
> {{ .to("projects/project/topics/topic")}}
> {{ *.withFactory(PubsubGrpcClient.FACTORY)*}}
> {{ .withTimestampAttribute("timestamp")}}{{;}}
> or
> {{PubsubIO.Read read = }}
> {{ PubsubIO.readMessages()}}
> {{ .fromSubscription("projects/project/subscriptions/name")}}
> {{ *.withFactory(PubsubGrpcClient.FACTORY)*}}
> {{ .withTimestampAttribute("timestamp");}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-5806) Allow to change the PubsubClientFactory when using PubsubIO
[
https://issues.apache.org/jira/browse/BEAM-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Logan HAUSPIE updated BEAM-5806:
Description:
By using the PubsubIO to read from or write to Pub/Sub we are obliged to use
the PubsubJsonClient to interact with the Pub/Sub API.
This PubsubJsonClient encode the message in base 64 and increase the size of
this one by 30% and there is no way to change the PubsubClient used by PubsubIO.
What I suggest is to allow developper to change the PubsubClientFactory by
specifying it at the definition-time like the following:
{{^PubsubIO.Read read = PubsubIO.readStrings()
.fromTopic(StaticValueProvider.of(topic))^}}
.withTimestampAttribute("myTimestamp")^}}
.withIdAttribute("myId")^}}
*.withClientFactory(PubsubGrpcClient.FACTORY)*;^}}
was:
By using the PubsubIO to read from or write to Pub/Sub we are obliged to use
the PubsubJsonClient to interact with the Pub/Sub API.
This PubsubJsonClient encode the message in base 64 and increase the size of
this one by 30% and there is no way to change the PubsubClient used by PubsubIO.
What I suggest is to allow developper to change the PubsubClientFactory by
specifying it at the definition-time like the following:
{{^PubsubIO.Read read =^ ^PubsubIO.readStrings()^}}
{{ ^.fromTopic(StaticValueProvider.of(topic))^}}
{{ ^.withTimestampAttribute("myTimestamp")^}}
{{ ^.withIdAttribute("myId")^}}
{{ ^*.withClientFactory(PubsubGrpcClient.FACTORY)*;^}}
> Allow to change the PubsubClientFactory when using PubsubIO
> ---
>
> Key: BEAM-5806
> URL: https://issues.apache.org/jira/browse/BEAM-5806
> Project: Beam
> Issue Type: New Feature
> Components: io-java-gcp
>Reporter: Logan HAUSPIE
>Assignee: Chamikara Jayalath
>Priority: Minor
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> By using the PubsubIO to read from or write to Pub/Sub we are obliged to use
> the PubsubJsonClient to interact with the Pub/Sub API.
> This PubsubJsonClient encode the message in base 64 and increase the size of
> this one by 30% and there is no way to change the PubsubClient used by
> PubsubIO.
>
> What I suggest is to allow developper to change the PubsubClientFactory by
> specifying it at the definition-time like the following:
> {{^PubsubIO.Read read = PubsubIO.readStrings()
> .fromTopic(StaticValueProvider.of(topic))^}}
> .withTimestampAttribute("myTimestamp")^}}
> .withIdAttribute("myId")^}}
> *.withClientFactory(PubsubGrpcClient.FACTORY)*;^}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-5806) Allow to change the PubsubClientFactory when using PubsubIO
[
https://issues.apache.org/jira/browse/BEAM-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Logan HAUSPIE updated BEAM-5806:
Description:
By using the PubsubIO to read from or write to Pub/Sub we are obliged to use
the PubsubJsonClient to interact with the Pub/Sub API.
This PubsubJsonClient encode the message in base 64 and increase the size of
this one by 30% and there is no way to change the PubsubClient used by PubsubIO.
What I suggest is to allow developper to change the PubsubClientFactory by
specifying it at the definition-time like the following:
{{^PubsubIO.Read read =^ ^PubsubIO.readStrings()^}}
{{ ^.fromTopic(StaticValueProvider.of(topic))^}}
{{ ^.withTimestampAttribute("myTimestamp")^}}
{{ ^.withIdAttribute("myId")^}}
{{ ^*.withClientFactory(PubsubGrpcClient.FACTORY)*;^}}
was:
By using the PubsubIO to read from or write to Pub/Sub we are obliged to use
the PubsubJsonClient to interact with the Pub/Sub API.
This PubsubJsonClient encode the message in base 64 and increase the size of
this one by 30% and there is no way to change the PubsubClient used by PubsubIO.
What I suggest is to allow developper to change the PubsubClientFactory by
specifying it at the definition-time like the following:
{{^PubsubIO.Read read =^}}
{{ ^PubsubIO.readStrings()^}}
{{ ^.fromTopic(StaticValueProvider.of(topic))^}}
{{ ^.withTimestampAttribute("myTimestamp")^}}
{{ ^.withIdAttribute("myId")^}}
{{ ^{{*.withClientFactory(PubsubGrpcClient.FACTORY)*}};^}}
> Allow to change the PubsubClientFactory when using PubsubIO
> ---
>
> Key: BEAM-5806
> URL: https://issues.apache.org/jira/browse/BEAM-5806
> Project: Beam
> Issue Type: New Feature
> Components: io-java-gcp
>Reporter: Logan HAUSPIE
>Assignee: Chamikara Jayalath
>Priority: Minor
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> By using the PubsubIO to read from or write to Pub/Sub we are obliged to use
> the PubsubJsonClient to interact with the Pub/Sub API.
> This PubsubJsonClient encode the message in base 64 and increase the size of
> this one by 30% and there is no way to change the PubsubClient used by
> PubsubIO.
>
> What I suggest is to allow developper to change the PubsubClientFactory by
> specifying it at the definition-time like the following:
> {{^PubsubIO.Read read =^ ^PubsubIO.readStrings()^}}
> {{ ^.fromTopic(StaticValueProvider.of(topic))^}}
> {{ ^.withTimestampAttribute("myTimestamp")^}}
> {{ ^.withIdAttribute("myId")^}}
> {{ ^*.withClientFactory(PubsubGrpcClient.FACTORY)*;^}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-5806) Allow to change the PubsubClientFactory when using PubsubIO
[
https://issues.apache.org/jira/browse/BEAM-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Logan HAUSPIE updated BEAM-5806:
Description:
By using the PubsubIO to read from or write to Pub/Sub we are obliged to use
the PubsubJsonClient to interact with the Pub/Sub API.
This PubsubJsonClient encode the message in base 64 and increase the zise of
this one by 30% and there is no way to change the PubsubClient used by PubsubIO.
What I suggest is to allow developper to change the PubsubClientFactory by
specifying it at the definition-time like the following:
```
PubsubIO.Read read =
PubsubIO.readStrings()
.fromTopic(StaticValueProvider.of(topic))
.withFactory(PubsubGrpcClient.FACTORY)
.withTimestampAttribute("myTimestamp")
.withIdAttribute("myId");
```
> Allow to change the PubsubClientFactory when using PubsubIO
> ---
>
> Key: BEAM-5806
> URL: https://issues.apache.org/jira/browse/BEAM-5806
> Project: Beam
> Issue Type: New Feature
> Components: io-java-gcp
>Reporter: Logan HAUSPIE
>Assignee: Chamikara Jayalath
>Priority: Minor
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> By using the PubsubIO to read from or write to Pub/Sub we are obliged to use
> the PubsubJsonClient to interact with the Pub/Sub API.
> This PubsubJsonClient encode the message in base 64 and increase the zise of
> this one by 30% and there is no way to change the PubsubClient used by
> PubsubIO.
>
> What I suggest is to allow developper to change the PubsubClientFactory by
> specifying it at the definition-time like the following:
> ```
> PubsubIO.Read read =
> PubsubIO.readStrings()
> .fromTopic(StaticValueProvider.of(topic))
> .withFactory(PubsubGrpcClient.FACTORY)
> .withTimestampAttribute("myTimestamp")
> .withIdAttribute("myId");
> ```
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-5806) Allow to change the PubsubClientFactory when using PubsubIO
Logan HAUSPIE created BEAM-5806: --- Summary: Allow to change the PubsubClientFactory when using PubsubIO Key: BEAM-5806 URL: https://issues.apache.org/jira/browse/BEAM-5806 Project: Beam Issue Type: New Feature Components: io-java-gcp Reporter: Logan HAUSPIE Assignee: Chamikara Jayalath -- This message was sent by Atlassian JIRA (v7.6.3#76005)
