[ https://issues.apache.org/jira/browse/AIRFLOW-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209792#comment-16209792 ]
ASF subversion and git services commented on AIRFLOW-1718: ---------------------------------------------------------- Commit 6078e753aac35aa4f5971a719d9f736c35396770 in incubator-airflow's branch refs/heads/master from [~cjqian] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=6078e75 ] [AIRFLOW-1718] Set num_retries on Dataproc job request execution Closes #2696 from cjqian/1718 > Increase num_retries polling value on Dataproc hook > --------------------------------------------------- > > Key: AIRFLOW-1718 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1718 > Project: Apache Airflow > Issue Type: Bug > Reporter: Crystal Qian > Assignee: Crystal Qian > Priority: Minor > Original Estimate: 24h > Remaining Estimate: 24h > > Currently, `num_retries = 0` when execute() is called > (https://google.github.io/google-api-python-client/docs/epy/googleapiclient.http.HttpRequest-class.html#execute), > which causes intermittent 500 errors > (https://stackoverflow.com/questions/46522261/deadline-exceeded-when-airflow-runs-spark-jobs). > We should increase this to allow retries for internal Dataproc queries to > other services in the short-term; also seeing if the `num_retries` count can > be increased at the _google-api-python-client_ level in the long-term. -- This message was sent by Atlassian JIRA (v6.4.14#64029)