kaxil commented on code in PR #76975: URL: https://github.com/apache/airflow/pull/76975#discussion_r4623437675
########## providers/common/ai/docs/changelog.rst: ########## @@ -80,6 +80,57 @@ Changelog typed ``str``; in 1.1.0 it defaulted to ``None`` and was typed ``str | None``. This changes only the signature and the annotated type, not the behaviour. +.. note:: + ``@task.llm_batch`` now runs OpenAI and Anthropic batches through pydantic-ai's ``Model.batch_*`` + methods and adds a Gemini adapter (``google:<model>``, Gemini Developer API only; a Vertex + ``google-cloud:`` model id is refused with the fix in the message). This raises the + ``pydantic-ai-slim`` floor to 2.66.0, the ``openai`` extra's ``openai`` floor to 3.31.0 and the + ``anthropic`` extra's ``anthropic`` floor to 1.15.0, and adds ``google-genai`` to the ``google`` + extra. + + What changes for an existing task: + + - ``request_params`` and a request's ``params`` keep working. The sampling settings (``temperature``, + ``top_p``, ...) and the provider-prefixed settings (``openai_*``, + ``anthropic_*``) are typed, and every other key is still sent as a request field. A setting the + model would silently drop (``seed`` on Anthropic; ``seed``, ``top_k``, ``frequency_penalty``, + ``presence_penalty``, ``logit_bias`` and ``stop_sequences`` on OpenAI) now fails before + submission, and a key the operator manages (``model``, ``messages``, ``input``, ...) is ignored + with a warning. A Chat Completions key has to be written the Responses way on OpenAI, for example + ``openai_reasoning_effort`` instead of ``reasoning_effort``. + - OpenAI batches run on the Responses API (``/v1/responses``) instead of Chat Completions, because + pydantic-ai batches only its Responses model. An ``openai-chat:`` model id is built as + ``openai:`` and logged. A gateway that implements batches only for ``/v1/chat/completions`` stops + working; whether LiteLLM implements them for ``/v1/responses`` has not been verified. Azure OpenAI + still has no adapter. + - Structured output uses the model's native JSON-schema output instead of a forced tool call or + ``response_format``, except for models without it. + - A cancelled batch is recognized from its landed results, because Anthropic reports it as ended. + - OpenAI no longer recovers a batch whose submit response was never recorded by reading the batch's + metadata. Like Anthropic and Gemini, it falls through to ``on_orphaned_intent``, whose default + ``"resubmit"`` submits a new batch. + - ``completion_window`` is ignored by every built-in adapter, and a value other than ``"24h"`` now logs + a warning. + - The state file records pydantic-ai's batch handle (state version 2). An Anthropic batch that an + earlier version of this provider submitted and is still running is re-attached; its state file is + left as it is. An OpenAI batch from an earlier version cannot be: it was created against Chat + Completions and its results cannot be read. The task fails with ``LLMBatchStateReadError`` + naming the batch; let it finish or cancel it at OpenAI, delete the task's state file and retry. + An outdated batch from an earlier version that was recorded under a different connection cannot + be cancelled automatically; the task log says so, and the batch can be cancelled at the provider. + - Anthropic and OpenAI results are streamed, so landing a batch holds only a small buffer of + responses in memory. Gemini results are still read through pydantic-ai, which downloads a batch's + results in one call, so a Gemini batch holds all of its response bodies in memory at once. Rows are still written one at a time. For a large + Gemini batch, size workers for it or split the input across mapped task instances with ``.expand()``. + - A task that was deferred by an earlier version of this provider and is still waiting when you + upgrade cannot poll its batch: the trigger reports an error naming the cause, and the task's retry + re-attaches to the batch. + - pydantic-ai deletes an OpenAI batch's uploaded input file only inside its own result collection, Review Comment: Took it out of the changelog. The OpenAI section of `llm_batch.rst` keeps the note and says the old adapter left the file behind too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
