Thanks folks, I did take a look at the change and swapping to the google-cloud-* libraries looks like a major refactor.
I did get something working by adding a shim so that the existing vendored libraries can work with the google-auth credential objects, similar to Luke's second point. Is this partial upgrade something that could be considered? Draft PR: https://github.com/apache/beam/pull/15004 Thanks! Chuck On Fri, Jun 11, 2021 at 11:06 AM Valentyn Tymofieiev <valen...@google.com> wrote: > > Thanks, Chuck for looking into this. We explored switching to google-cloud-* > python libraries for Dataflow runner purposes, and encountered several issues > related to dependency management of these libraries in Google-internal > repository, which were difficult to address just in Beam plane without > vendroing these dependencies. We are currently working to address these > issues internally before coming back to this effort. I would not recommend > switching these libraries on master branch without commitment/help from > Google Dataflow runner maintainers at this time, although we can certainly > explore what external (Beam) changes will be needed. > > On Thu, Jun 10, 2021 at 3:55 PM Luke Cwik <lc...@google.com> wrote: >> >> I did something very similar during the Dataflow Java 1.x to Beam Java 2.x >> migration. The work boiled down to: >> * swapping to a different library to get the application default credentials >> (including fixing upstream bugs at Google and improving some documentation) >> * swapping existing API calls to use the new credentials object (was easy >> since there was a trivial wrapper object that allowed you to convert new >> credentials object into the old type that some API client libraries only >> supported) >> * a bunch of documentation and trivial plumbing issues >> >> On Fri, May 14, 2021 at 5:33 PM Ahmet Altay <al...@google.com> wrote: >>> >>> +Valentyn Tymofieiev might have an idea. >>> >>> On Mon, May 3, 2021 at 4:12 PM Chuck Yang <chuck.y...@getcruise.com> wrote: >>>> >>>> Hi Beam devs, >>>> >>>> I saw there has been some previous discussion [1][2] around removing >>>> the deprecated oauth2client dependency and using the supported >>>> google-auth dependency instead. A portion of this work seems to >>>> involve migrating off of google-apitools since this call [3] is not >>>> supported by credentials objects emitted by google-auth. >>>> >>>> Does anyone have any experience/insights on how much work migrating >>>> off of oauth2client would involve? I might be able to help out but >>>> wanted to see a) if anyone is already looking at this and b) if there >>>> are any hidden obstacles beyond needing to move from google-apitools >>>> to the google-cloud-* libraries. Any pointers are appreciated! >>>> >>>> We're interested in this migration because of the need to use custom >>>> token URIs for issuing service account tokens--it's supported by >>>> google-auth but not oauth2client. >>>> >>>> [1] https://issues.apache.org/jira/browse/BEAM-7352 >>>> [2] https://github.com/google/apitools/issues/225#issuecomment-434884589 >>>> [3] >>>> https://github.com/google/apitools/blob/v0.5.31/apitools/base/py/base_api.py#L266 >>>> >>>> Thanks! >>>> Chuck >>>> >>>> -- >>>> >>>> >>>> *Confidentiality Note:* We care about protecting our proprietary >>>> information, confidential material, and trade secrets. This message may >>>> contain some or all of those things. Cruise will suffer material harm if >>>> anyone other than the intended recipient disseminates or takes any action >>>> based on this message. If you have received this message (including any >>>> attachments) in error, please delete it immediately and notify the sender >>>> promptly. -- *Confidentiality Note:* We care about protecting our proprietary information, confidential material, and trade secrets. This message may contain some or all of those things. Cruise will suffer material harm if anyone other than the intended recipient disseminates or takes any action based on this message. If you have received this message (including any attachments) in error, please delete it immediately and notify the sender promptly.
