I don't have enough context to answer all of the questions, but looking at PubsubIO it seems to use the official libraries, e.g. see Pubsub doc [1] vs Pubsub IO GRPC client [2]. Correct me if I misunderstood your question.
[1] https://cloud.google.com/pubsub/docs/publisher#pubsub-publish-message-java [2] https://github.com/apache/beam/blob/2e759fecf63d62d110f29265f9438128e3bdc8ab/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubGrpcClient.java#L189 Pubsub IO JSON client seems to use a slightly different approach but still relies on somewhat official path, e.g. Pubsub doc [3] (javadoc[4]) vs Pubsub IO JSON client [5]. [3] https://developers.google.com/api-client-library/java/apis/pubsub/v1 [4] https://developers.google.com/resources/api-libraries/documentation/pubsub/v1/java/latest/com/google/api/services/pubsub/Pubsub.html [5] https://github.com/apache/beam/blob/2e759fecf63d62d110f29265f9438128e3bdc8ab/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubJsonClient.java#L130 The latter seems to be the older library, so I would assume it's for legacy reasons. Regards, Anton On Wed, Jan 2, 2019 at 9:03 AM Jeff Klukas <jklu...@mozilla.com> wrote: > I'm building a high-volume Beam pipeline using PubsubIO and running into > some concerns over performance and delivery semantics, prompting me to want > to better understand the implementation. Reading through the library, > PubsubIO appears to be a completely separate implementation of Pubsub > client behavior from Google's own Java client. As a developer trying to > read and understand the implementation, this is a significant hurdle, since > any previous knowledge of the Google library is not applicable and is > potentially at odds with what's in PubsubIO. > > Why doesn't beam use the Google clients for PubsubIO, BigQueryIO, etc.? Is > it for historical reasons? Is there difficulty in packaging and integration > of the Google clients? Or are the needs for Beam just substantially > different from what the Google libraries provide? >
