My apologies. I got the terminology entirely wrong. As you say, PubsubIO and other Beam components _do_ use the official Google API client library (google-api-client). They do not, however, use the higher-level Google Cloud libraries such as google-cloud-pubsub which provide abstractions on top of the API client library.
I am wondering whether there are technical reasons not to use the higher-level service-specific libraries, or whether this is simply historical. On Wed, Jan 2, 2019 at 12:38 PM Anton Kedin <ke...@google.com> wrote: > I don't have enough context to answer all of the questions, but looking at > PubsubIO it seems to use the official libraries, e.g. see Pubsub doc [1] > vs Pubsub IO GRPC client [2]. Correct me if I misunderstood your question. > > [1] > https://cloud.google.com/pubsub/docs/publisher#pubsub-publish-message-java > [2] > https://github.com/apache/beam/blob/2e759fecf63d62d110f29265f9438128e3bdc8ab/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubGrpcClient.java#L189 > > Pubsub IO JSON client seems to use a slightly different approach but still > relies on somewhat official path, e.g. Pubsub doc [3] (javadoc[4]) vs > Pubsub IO JSON client [5]. > > [3] https://developers.google.com/api-client-library/java/apis/pubsub/v1 > [4] > https://developers.google.com/resources/api-libraries/documentation/pubsub/v1/java/latest/com/google/api/services/pubsub/Pubsub.html > [5] > https://github.com/apache/beam/blob/2e759fecf63d62d110f29265f9438128e3bdc8ab/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubJsonClient.java#L130 > > The latter seems to be the older library, so I would assume it's for > legacy reasons. > > Regards, > Anton > > > On Wed, Jan 2, 2019 at 9:03 AM Jeff Klukas <jklu...@mozilla.com> wrote: > >> I'm building a high-volume Beam pipeline using PubsubIO and running into >> some concerns over performance and delivery semantics, prompting me to want >> to better understand the implementation. Reading through the library, >> PubsubIO appears to be a completely separate implementation of Pubsub >> client behavior from Google's own Java client. As a developer trying to >> read and understand the implementation, this is a significant hurdle, since >> any previous knowledge of the Google library is not applicable and is >> potentially at odds with what's in PubsubIO. >> >> Why doesn't beam use the Google clients for PubsubIO, BigQueryIO, etc.? >> Is it for historical reasons? Is there difficulty in packaging and >> integration of the Google clients? Or are the needs for Beam just >> substantially different from what the Google libraries provide? >> >
