I think you need some support from the underlying GCS or HTTP client libraries to do this.
All calls to GCS are done through gcsio library: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/gcsio.py Object insert requests to GCS are created here: https://github.com/apache/beam/blob/79c1932af6df31c677676a43afd6f68406225686/sdks/python/apache_beam/io/gcp/gcsio.py#L647 gcsio uses a client generated by apitools client library today, I'm not sure if this provides sufficient support track HTTP requests if that's what you are looking for. The latest GCS Python client library seems to be following. So might be be good to look into whether that provides better support for what you are looking for as well (but changing client libraries will be a larger task). https://pypi.org/project/google-cloud-storage/ https://github.com/googleapis/python-storage Thanks, Cham On Mon, May 24, 2021 at 6:10 AM Miguel Hernández Sandoval < [email protected]> wrote: > Sorry, forgot to include the ticket > [1] https://issues.apache.org/jira/browse/BEAM-11984 > > On Mon, May 24, 2021 at 7:54 AM Miguel Hernández Sandoval < > [email protected]> wrote: > >> Hi team, >> I've been working on this ticket[1] where some API requests to GCS are >> counted and kept as process-wide metrics. >> >> I was hoping you could give me some advice on how to test this kind of >> feature to test it using a real response from GCS instead of mocked >> responses and how to test it using a pipeline. >> >> Thank you all for your help >> -Mike >> > > > -- > > Miguel Hernández Sandoval | WIZELINE > > Software Engineer > > [email protected] > > Amado Nervo 2200, Esfera P6, Col. Jardines del Sol, 45050 Zapopan, Jal. > > > > > > > > > *This email and its contents (including any attachments) are being sent > toyou on the condition of confidentiality and may be protected by > legalprivilege. Access to this email by anyone other than the intended > recipientis unauthorized. If you are not the intended recipient, please > immediatelynotify the sender by replying to this message and delete the > materialimmediately from your system. Any further use, dissemination, > distributionor reproduction of this email is strictly prohibited. Further, > norepresentation is made with respect to any content contained in this > email.*
