FrankYang0529 commented on code in PR #75649:
URL: https://github.com/apache/airflow/pull/75649#discussion_r4408391229
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/msgraph.py:
##########
@@ -676,6 +676,59 @@ async def assert_allowed_host(self, url: str | None) ->
None:
f"Microsoft Graph endpoints {sorted(allowed_netlocs)}."
)
+ async def iter_paginated_run(
+ self,
+ url: str = "",
+ response_type: str | None = None,
+ path_parameters: dict[str, Any] | None = None,
+ method: str = "GET",
+ query_parameters: dict[str, Any] | None = None,
+ headers: dict[str, str] | None = None,
+ data: dict[str, Any] | str | BytesIO | None = None,
+ pagination_function: PaginationCallable | None = None,
+ ) -> AsyncIterator[Any]:
+ """
+ Yield the pages of a paginated request one at a time.
+
+ The caller handles each page before the next one is requested, so a
collection of any size passes
+ through in page-sized steps instead of the single list
``paginated_run`` builds. The next page is
+ resolved by ``pagination_function`` (``default_pagination`` by
default: ``@odata.nextLink``, or
+ ``$skip`` on counted queries) and its host is checked with
``assert_allowed_host`` before it is
+ fetched. ``pagination_function`` receives the same arguments as in
``paginated_run`` except
+ ``responses``: nothing keeps the pages already yielded.
+ """
+ if pagination_function is None:
+ pagination_function = self.default_pagination
+
+ while url:
+ response = await self.run(
+ url=url,
+ response_type=response_type,
+ path_parameters=path_parameters,
+ method=method,
+ query_parameters=query_parameters,
+ headers=headers,
+ data=data,
+ )
+
+ if not response:
+ return
+
+ yield response
Review Comment:
The page yielded here is the same dict that `pagination_function` receives a
few lines below. The `iter_paginated_run` generator calls `pagination_function`
after the caller's loop body has finished with the page. So when the caller
changes the page, `default_pagination` decides the next request by reading the
changed page. Do we need to mention that the caller must not modify the yielded
page in docstring?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
