VladaZakharova opened a new pull request, #75969:
URL: https://github.com/apache/airflow/pull/75969
1. CreateBatchPredictionJobOperator (BatchPredictionJobHook)
Using region="global" fails because of how the google-cloud-aiplatform
library builds the endpoint. When the location is passed directly to
BatchPredictionJob.create / submit, the library always builds
{location}-aiplatform.googleapis.com. For global that gives
global-aiplatform.googleapis.com, which doesn't exist. The library only uses
the correct aiplatform.googleapis.com when the location is set with
aiplatform.init(), which is why the customer's workaround works.
We also found a second problem in Airflow's hook: for multi-regional
locations such as us, it built us-aiplatform.googleapis.com instead of
aiplatform.us.rep.googleapis.com. That broke checking the job status after
creation (get, list and delete, plus deferrable mode).
Fix: the hook now picks the correct endpoint for each type of location:
global → aiplatform.googleapis.com
us / eu → aiplatform.{location}.rep.googleapis.com
regional → {region}-aiplatform.googleapis.com
It passes this endpoint to aiplatform.init() before every create/submit
call, and uses the same endpoint for all other calls. Regional endpoints keep
working as before.
Limitation: the current google-cloud-aiplatform version doesn't accept eu as
a location at all, so eu still can't be used with this operator. For eu, use
the GenAI operator below.
2. GenAIGeminiCreateBatchJobOperator (GenAIGeminiAPIHook)
Confirmed. The hook always created the google-genai client in Gemini
Developer API mode (vertexai=False, API key). That mode only accepts inline
requests or Gemini Files API files, so GCS and BigQuery input and output
weren't possible.
Fix: we added an optional vertexai parameter (default False, so existing
users aren't affected). With vertexai=True, the operator uses Gemini on Vertex
AI with the Airflow Google Cloud connection, project_id and location. GCS
(gs://) and BigQuery (bq://) input and output then work. location can be
global, us, eu or a region such as us-central1. When the job finishes, the GCS
or BigQuery result location is pushed to XCom. The operator also now fails
early with a clear error if the settings don't match the chosen mode, for
example a GCS input without vertexai=True.
So, now:
- CreateBatchPredictionJobOperator: works with global, us and regional
locations (not eu, because of the library limitation).
- GenAIGeminiCreateBatchJobOperator with vertexai=True: works with
GCS/BigQuery input and output on global, us, eu and regional locations.
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
For user-facing UI changes, please attach before/after screenshots (or a
short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
