[
https://issues.apache.org/jira/browse/BEAM-7738?focusedWorklogId=290210&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-290210
]
ASF GitHub Bot logged work on BEAM-7738:
----------------------------------------
Author: ASF GitHub Bot
Created on: 07/Aug/19 04:29
Start Date: 07/Aug/19 04:29
Worklog Time Spent: 10m
Work Description: chadrik commented on pull request #9268: [BEAM-7738]
Add external transform support to PubSubIO
URL: https://github.com/apache/beam/pull/9268#discussion_r311364462
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java
##########
@@ -674,6 +680,85 @@ public String toString() {
abstract Builder<T> setClock(@Nullable Clock clock);
abstract Read<T> build();
+
+ @Override
+ public PTransform<PBegin, PCollection<T>>
buildExternal(External.Configuration config) {
+ if (config.topic != null) {
+ StaticValueProvider<String> topic =
StaticValueProvider.of(utf8String(config.topic));
+ setTopicProvider(NestedValueProvider.of(topic, new
TopicTranslator()));
+ }
+ if (config.subscription != null) {
+ StaticValueProvider<String> subscription =
+ StaticValueProvider.of(utf8String(config.subscription));
+ setSubscriptionProvider(
+ NestedValueProvider.of(subscription, new
SubscriptionTranslator()));
+ }
+ if (config.idAttribute != null) {
+ String idAttribute = utf8String(config.idAttribute);
+ setIdAttribute(idAttribute);
+ }
+ if (config.timestampAttribute != null) {
+ String timestampAttribute = utf8String(config.timestampAttribute);
+ setTimestampAttribute(timestampAttribute);
+ }
+ setNeedsAttributes(config.needsAttributes);
+ setPubsubClientFactory(FACTORY);
+ if (config.needsAttributes) {
+ SimpleFunction<PubsubMessage, T> parseFn =
+ (SimpleFunction<PubsubMessage, T>) new IdentityMessageFn();
+ setParseFn(parseFn);
+ // FIXME: call setCoder(). need to use PubsubMessage proto to be
compatible with python
Review comment:
I serialized the `PubsubMessage` using protobufs. Since there's no
cross-language coder for `PubsubMessage`, and I assumed it would be overreach
to add one, I used the bytes coder and then handled converting to and from
protobufs in code that lives close to the transforms.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 290210)
Time Spent: 1h (was: 50m)
> Support PubSubIO to be configured externally for use with other SDKs
> --------------------------------------------------------------------
>
> Key: BEAM-7738
> URL: https://issues.apache.org/jira/browse/BEAM-7738
> Project: Beam
> Issue Type: New Feature
> Components: io-java-gcp, runner-flink, sdk-py-core
> Reporter: Chad Dombrova
> Assignee: Chad Dombrova
> Priority: Major
> Labels: portability
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Now that KafkaIO is supported via the external transform API (BEAM-7029) we
> should add support for PubSub.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
