mahesh989 opened a new issue, #45877: URL: https://github.com/apache/airflow/issues/45877
### Description The `DataplexRunDataQualityScanOperator` and `DataplexGetDataQualityScanResultOperator` are now in non-deferrable mode during operation. This results in the operators using resources excessively when doing lengthy Dataplex API operations. Waiting for API answers causes inefficient use of resources and slows down the completion of other actions in workflows. Adding a **deferrable mode** to these operators will allow them to pause execution during wait periods, freeing up resources for other tasks. This change will optimize resource utilization and improve overall workflow performance. ### Use case/motivation ### Use Case 1. ** Efficient Long-Running Workflows**: In workflows where tasks interact with the Dataplex API for extended durations, deferrable mode will significantly reduce resource usage, enabling better performance. 2. **Scalable Workflows**: Deferrable mode will help users scale their workflows by making better use of available infrastructure. ### Motivation - **Pause and Free Resources** while waiting for the Dataplex API to complete its operations. - **Seamlessly Resume Execution** once the API task is complete, without any manual intervention. ### Related issues There doesn't appear to be another linked issue regarding functionality addressed by this PR. ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
