tooptoop4 commented on a change in pull request #6909: [AIRFLOW-6359] spark_submit_hook.py status polling interval config URL: https://github.com/apache/airflow/pull/6909#discussion_r361841452
########## File path: airflow/contrib/hooks/spark_submit_hook.py ########## @@ -492,8 +496,8 @@ def _start_driver_status_tracking(self): while self._driver_status not in ["FINISHED", "UNKNOWN", "KILLED", "FAILED", "ERROR"]: - # Sleep for 1 second as we do not want to spam the cluster - time.sleep(1) + # Sleep for n seconds as we do not want to spam the cluster + time.sleep(self._status_poll_interval) Review comment: I know what you mean but this is better than existing hardcoding, even changing to sleep 10 will reduce load on spark cluster ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services