shashbha14 opened a new pull request, #62674:
URL: https://github.com/apache/airflow/pull/62674
Summary
This patch for the `google.cloud.sql` module for Airflow allows for only one
administrative operation at a time for an instance (import, export, clone,
patch, etc.). When multiple operations are executed concurrently, the API
returns HTTP 409 with the reason `operation_in_progress`. This patch includes
an orchestration-level sensor as well as an operator-level retry, so that the
problem is addressed by the DAGs themselves, avoiding the need for workarounds.
Fixes #62671
Changes
### Option A: Deferrable sensor (orchestration)
- **CloudSQLInstanceOperationsSensor**: This sensor will wait until there is
no PENDING/RUNNING operation running on the instance. This is instance-aware,
so it will work across multiple DAGs.
- **CloudSQLInstanceOperationsTrigger**: This is an async trigger, which is
used by the sensor.
- **list_instance_operations**: This is a new method for `CloudSQLHook` as
well as `CloudSQLAsyncHook`, which is used to list the operations for an
instance, using the Operations API.
### Option B: Operator-level retry
- **@operation_in_progress_retry()** is applied to `import_instance`,
`export_instance`, `clone_instance` methods of `CloudSQLHook`. This will retry
the call with exponential backoff when the HTTP 409 status is returned with the
status `operation_in_progress`, as is done with `GoogleBaseHook`.
### Other
- The sensor is added to the provider info, as well as unit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
