dabla opened a new issue, #74089:
URL: https://github.com/apache/airflow/issues/74089
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.3.2 (installed with the `constraints-3-3` constraints)
### What happened and how to reproduce it?
With `microsoft-kiota-http` >= 1.13 (the `constraints-3-3` branch pins
1.14.0, `constraints-main` 1.14.1), every request made through
`KiotaRequestAdapterHook` bypasses the whole Microsoft Graph middleware chain.
The frozen `constraints-3.3.2` still pins 1.12.3 and is not affected.
**Cause:** `microsoft-kiota-http` 1.13.0 stopped setting `request.options`
(`setattr(request, "options", request_options)` in 1.12.x) and passes the
request options in `request.extensions[REQUEST_OPTIONS_KEY]` instead.
`msgraph-core`'s `AsyncGraphTransport.handle_async_request` (1.5.1, the latest
release) only runs its middleware pipeline when `hasattr(request, "options")`:
```python
async def handle_async_request(self, request: httpx.Request) ->
httpx.Response:
if self.pipeline and hasattr(request, 'options'):
self.set_request_context_and_feature_usage(request)
response = await self.pipeline.send(request)
return response
response = await self.transport.handle_async_request(request)
return response
```
So the request goes straight to the httpx transport, without the
`RedirectHandler`, `RetryHandler`, `ParametersNameDecodingHandler`,
`UrlReplaceHandler`, `UserAgentHandler`, `HeadersInspectionHandler` or
`GraphTelemetryHandler`.
**Effects:**
- Redirects are not followed. Downloading a drive item
(`drives/{drive-id}/items/{item-id}/content`, which answers `302` to a
pre-authenticated SharePoint URL) returns Graph's raw `302` with an empty body:
`KiotaRequestAdapterHook.run(url=..., response_type="bytes")` returns `None`.
- 429 and 503 with `Retry-After` are no longer retried by Kiota's
`RetryHandler`.
- Calls answering `200` directly keep working, which makes the problem easy
to miss. This affects `KiotaRequestAdapterHook`, `MSGraphAsyncOperator`,
`MSGraphSensor` and the Power BI hooks/operators built on them.
**Reproduce** (no credentials needed; `httpx.MockTransport` plays Graph and
SharePoint):
```python
import asyncio
import httpx
from kiota_http.middleware import REQUEST_OPTIONS_KEY # exists since
microsoft-kiota-http 1.13
from msgraph_core import GraphClientFactory
def answer(request: httpx.Request) -> httpx.Response:
if request.url.host == "graph.microsoft.com":
return httpx.Response(302, headers={"location":
"https://contoso.sharepoint.com/download.aspx"})
return httpx.Response(200, content=b"%PDF-1.7")
async def main():
client = GraphClientFactory.create_with_default_middleware(
client=httpx.AsyncClient(transport=httpx.MockTransport(answer))
)
async with client:
# How microsoft-kiota-http >= 1.13 builds the request: options in
extensions only.
request = httpx.Request(
"GET",
"https://graph.microsoft.com/v1.0/drives/d/items/i/content";,
extensions={REQUEST_OPTIONS_KEY: {}},
)
response = await client.send(request)
print(response.status_code) # 302: the RedirectHandler never ran
asyncio.run(main())
```
With `microsoft-kiota-http==1.14.0` and `msgraph-core==1.5.1` this prints
`302`. With the request options also set as `request.options`, which is what
kiota-http 1.12.x did, the redirect is followed and it prints `200`.
### What you think should happen instead?
The Graph middleware should run whatever `microsoft-kiota-http` 1.x version
the provider's dependency range allows (`microsoft-kiota-http>=1.9.4,<2.0.0`):
redirects followed, 429/503 retried.
Until `msgraph-core` reads the options from `request.extensions`, the
provider could either:
1. cap `microsoft-kiota-http<1.13` (and keep the other `microsoft-kiota-*`
packages on the matching 1.12.x), or
2. hand the options back to the transport in `KiotaRequestAdapterHook`, e.g.
wrap the transport so that a request without `options` but with
`REQUEST_OPTIONS_KEY` in `extensions` gets `request.options =
request.extensions[REQUEST_OPTIONS_KEY]` before `AsyncGraphTransport` checks
for it.
Option 2 keeps the newer Kiota releases usable. We run it as a plugin patch,
and it is verified against a real `302` → `200` redirect. The root cause also
deserves an issue on `microsoftgraph/msgraph-sdk-python-core`.
### Operating System
Fedora 44 (container image)
### Deployment
Other Docker-based deployment
### Apache Airflow Provider(s)
microsoft-azure
### Versions of Apache Airflow Providers
apache-airflow-providers-microsoft-azure==15.1.0
microsoft-kiota-http==1.14.0
microsoft-kiota-abstractions==1.14.0
microsoft-kiota-authentication-azure==1.14.0
msgraph-core==1.5.1
httpx==0.28.1
### Anything else?
It happens on every request, not intermittently. Unit tests do not catch it
because they mock the request adapter instead of sending a request through
`AsyncGraphTransport`. A regression test should send a `302` through
`GraphClientFactory.create_with_default_middleware` with an
`httpx.MockTransport`, as in the reproduction above.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
