[ https://issues.apache.org/jira/browse/AIRFLOW-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006090#comment-17006090 ]
ASF subversion and git services commented on AIRFLOW-6359: ---------------------------------------------------------- Commit 6cc1123e6f2ed09b0fa42fa85fcac97a06ba0924 in airflow's branch refs/heads/master from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=6cc1123 ] [AIRFLOW-6359] Make Spark status_poll_interval explicit (#6978) > spark_submit_hook.py polling too frequent > ----------------------------------------- > > Key: AIRFLOW-6359 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6359 > Project: Apache Airflow > Issue Type: Improvement > Components: hooks > Affects Versions: 1.10.6 > Reporter: t oo > Assignee: t oo > Priority: Major > Fix For: 1.10.8 > > > spark polling every second is too frequent, use config to change to desired > interval, ie 15 seconds > contrib/hooks/spark_submit_hook.py Poll spark server at a custom interval > instead of every second > > BEFORE > Sleep for 1 second as we do not want to spam the cluster > time.sleep(1) > > AFTER > import airflow > from airflow import configuration as conf > Sleep for n second as we do not want to spam the cluster > _poll_interval = conf.getint('sparksubmit', 'poll_interval') > time.sleep(_poll_interval) -- This message was sent by Atlassian Jira (v8.3.4#803005)