[ https://issues.apache.org/jira/browse/AIRFLOW-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tomasz Urbaszek resolved AIRFLOW-6359. -------------------------------------- Resolution: Fixed > spark_submit_hook.py polling too frequent > ----------------------------------------- > > Key: AIRFLOW-6359 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6359 > Project: Apache Airflow > Issue Type: Improvement > Components: hooks > Affects Versions: 1.10.6 > Reporter: t oo > Assignee: t oo > Priority: Major > > spark polling every second is too frequent, use config to change to desired > interval, ie 15 seconds > contrib/hooks/spark_submit_hook.py Poll spark server at a custom interval > instead of every second > > BEFORE > Sleep for 1 second as we do not want to spam the cluster > time.sleep(1) > > AFTER > import airflow > from airflow import configuration as conf > Sleep for n second as we do not want to spam the cluster > _poll_interval = conf.getint('sparksubmit', 'poll_interval') > time.sleep(_poll_interval) -- This message was sent by Atlassian Jira (v8.3.4#803005)