jigneshjain25 commented on PR #74753: URL: https://github.com/apache/airflow/pull/74753#issuecomment-6295718044
Thanks for the review @olegkachur-e! All feedback has been addressed in commit `4bb6288df1` on `dataproc-ho-support`: 1. **Removed custom classes & redundant validation**: Completely removed `FlexibleInstanceGroupConfig`, `HardwareClass` enum, `flexible_scale_cluster_update`, and client-side validation methods from `dataproc.py`. Airflow Dataproc operators now act as pass-through orchestrators passing `cluster_config`, `cluster`, and `update_mask` directly to the backend API via `DataprocHook`. Users can provide plain dictionaries or native `google.cloud.dataproc_v1` proto types (`FlexibleInstanceGroupConfig`). 2. **Cleaned up operator docstring**: Removed the inline HO example block from `DataprocCreateClusterOperator`'s `cluster_config` docstring, referencing the documentation guide and example DAG instead. 3. **Official documentation links**: Added direct links in `dataproc.rst` to the official Google Cloud documentation for [Dataproc High Obtainability mode](https://cloud.google.com/dataproc/docs/concepts/compute/high-obtainability) and the [FlexibleInstanceGroupConfig API reference](https://cloud.google.com/dataproc/docs/reference/rest/v1/ClusterConfig#FlexibleInstanceGroupConfig). 4. **Updated `uv.lock`**: Refreshed `uv.lock` with `google-cloud-dataproc>=5.32.0` and exact PyPI release metadata and hashes. 5. **Minimized example DAG resources**: Updated `example_dataproc_cluster_high_obtainability.py` to use minimal resources (2 cores, 32GB boot disk per node across master, primary workers, and secondary worker), plain dictionaries, and consolidated scaling into a single `DataprocUpdateClusterOperator` call scaling `worker_flexible_config` to 3 and `secondary_worker_flexible_config` to 0. 6. **Verified DAG execution & test suite**: - The example DAG was executed and passed end-to-end on GCP test project `dataproc-test-jnj` (create -> scale -> spark_pi -> delete all `SUCCESS`). - Ran `ruff check providers/google`, `ruff format --check providers/google`, and `pytest` on both `test_dataproc_high_obtainability.py` and `test_dataproc.py` (all tests passing). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
