Elad created AIRFLOW-5328: ----------------------------- Summary: EmrCreateJobFlowOperator success when EMR isn't ready Key: AIRFLOW-5328 URL: https://issues.apache.org/jira/browse/AIRFLOW-5328 Project: Apache Airflow Issue Type: Bug Components: aws Affects Versions: 1.10.4 Reporter: Elad
EmrCreateJobFlowOperator returns almost instantly. It shouldn't return until the Emr cluster id ready. It doesn't make sense to submit tasks to EMR when the cluster isn't ready. Currently, tasks are submitted and waiting until the cluster will finish its initialization. This can be checked with Waiter: [https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/emr.html] something like: {code:java} waiter = client.get_waiter('cluster_running').wait(ClusterId=response['JobFlowId'], WaiterConfig={'Delay': 32, 'MaxAttempts': 60}){code} -- This message was sent by Atlassian Jira (v8.3.2#803003)