mik-laj opened a new pull request #11735: URL: https://github.com/apache/airflow/pull/11735
Some tasks cannot be performed simultaneously, resulting in instability. Sometimes this DAG was successful but sometimes unsuccessful. Before:  After:  Logs: ``` [2020-10-22 00:23:59,786] {cloud_memorystore.py:310} INFO - Failovering Instance: projects/polidea-airflow/locations/europe-north1/instances/test-memorystore-redis-2 [2020-10-22 00:24:00,358] {taskinstance.py:1337} ERROR - 400 Unable to failover due to ongoing operations. Please wait until instance is in READY state. com.google.apps.framework.request.StatusException: <eye3 title='FAILED_PRECONDITION'/> generic::FAILED_PRECONDITION: Unable to failover due to ongoing operations. Please wait until instance is in READY state. Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable return callable_(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 826, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.FAILED_PRECONDITION details = "Unable to failover due to ongoing operations. Please wait until instance is in READY state. com.google.apps.framework.request.StatusException: <eye3 title='FAILED_PRECONDITION'/> generic::FAILED_PRECONDITION: Unable to failover due to ongoing operations. Please wait until instance is in READY state." debug_error_string = "{"created":"@1603326240.357049200","description":"Error received from peer ipv4:216.58.215.106:443","file":"src/core/lib/surface/call.cc","file_line":1061,"grpc_message":"Unable to failover due to ongoing operations. Please wait until instance is in READY state.\ncom.google.apps.framework.request.StatusException: <eye3 title='FAILED_PRECONDITION'/> generic::FAILED_PRECONDITION: Unable to failover due to ongoing operations. Please wait until instance is in READY state.","grpc_status":9}" > The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/airflow/airflow/models/taskinstance.py", line 1076, in _run_raw_task self._prepare_and_execute_task_with_callbacks(context, task) File "/opt/airflow/airflow/models/taskinstance.py", line 1198, in _prepare_and_execute_task_with_callbacks result = self._execute_task(context, task_copy) File "/opt/airflow/airflow/models/taskinstance.py", line 1243, in _execute_task result = task_copy.execute(context=context) File "/opt/airflow/airflow/providers/google/cloud/operators/cloud_memorystore.py", line 409, in execute metadata=self.metadata, File "/opt/airflow/airflow/providers/google/common/hooks/base_google.py", line 373, in inner_wrapper return func(self, *args, **kwargs) File "/opt/airflow/airflow/providers/google/cloud/hooks/cloud_memorystore.py", line 317, in failover_instance metadata=metadata, File "/usr/local/lib/python3.6/site-packages/google/cloud/redis_v1/gapic/cloud_redis_client.py", line 939, in failover_instance request, retry=retry, timeout=timeout, metadata=metadata File "/usr/local/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__ return wrapped_func(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable six.raise_from(exceptions.from_grpc_error(exc), exc) File "<string>", line 3, in raise_from google.api_core.exceptions.FailedPrecondition: 400 Unable to failover due to ongoing operations. Please wait until instance is in READY state. com.google.apps.framework.request.StatusException: <eye3 title='FAILED_PRECONDITION'/> generic::FAILED_PRECONDITION: Unable to failover due to ongoing operations. Please wait until instance is in READY state. [2020-10-22 00:24:00,370] {taskinstance.py:1381} INFO - Marking task as FAILED. dag_id=gcp_cloud_memorystore_redis, task_id=failover-instance, execution_date=20201021T000000, start_date=20201022T002358, end_date=20201022T002400 [2020-10-22 00:24:00,728] {local_task_job.py:106} INFO - Task exited with return code 1 ``` <!-- Thank you for contributing! Please make sure that your code changes are covered with tests. And in case of new features or big changes remember to adjust the documentation. Feel free to ping committers for the review! In case of existing issue, reference it using one of the following: closes: #ISSUE related: #ISSUE How to write a good git commit message: http://chris.beams.io/posts/git-commit/ --> --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
