alamashir opened a new issue, #65467:
URL: https://github.com/apache/airflow/issues/65467
### Description
Add first-class operators for managing BigQuery routines — scalar UDFs,
stored procedures, table-valued functions, aggregate functions, remote
functions, and Spark stored procedures — in the Google provider.
Today the only way to deploy a BigQuery routine from Airflow is to embed raw
`CREATE FUNCTION` / `CREATE PROCEDURE` DDL inside `BigQueryInsertJobOperator`.
The `google-cloud-bigquery` client already exposes `create_routine`,
`get_routine`, `update_routine`, `delete_routine`, and `list_routines`, but
`BigQueryHook` does not wrap them.
Proposed new operators in
`airflow.providers.google.cloud.operators.bigquery`:
- `BigQueryCreateRoutineOperator` — supports every routine type;
`if_exists={"skip","replace","fail"}`
- `BigQueryUpdateRoutineOperator`
- `BigQueryDeleteRoutineOperator`
- `BigQueryGetRoutineOperator`
- `BigQueryListRoutinesOperator`
New sensor in `airflow.providers.google.cloud.sensors.bigquery`:
- `BigQueryRoutineExistenceSensor`
Plus five matching hook methods in `BigQueryHook`. No new provider
dependency — everything is already shipped with `google-cloud-bigquery`.
### Use case/motivation
Deploying a routine and a pipeline that depends on it today requires either
(a) hand-rolled DDL in `BigQueryInsertJobOperator` with no idempotency beyond
`CREATE OR REPLACE`, no typed validation, and no structured deployment record,
or (b) managing routines out-of-band via Terraform (`google_bigquery_routine`)
or dbt hacks — which splits routine deployment from the pipeline that consumes
it.
First-class operators let a single DAG own both the routine definition and
the pipeline that uses it, with idempotent re-runs and typed kwargs that
validate at author time instead of at job execution. Remote functions (BQ →
Cloud Functions/Cloud Run) and Spark stored procedures (BQ → Dataproc
Serverless) particularly benefit, since both tightly couple routine deployment
to other infra.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
