[jira] [Commented] (AIRFLOW-5544) GCP datastore operation pull should have some retry to avoid false alarm
[
https://issues.apache.org/jira/browse/AIRFLOW-5544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982955#comment-16982955
]
Yang Hu commented on AIRFLOW-5544:
--
Hi [~kamil.bregula]
What improvements are there? Do you have a link or version that we can test?
> GCP datastore operation pull should have some retry to avoid false alarm
>
>
> Key: AIRFLOW-5544
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5544
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp
>Affects Versions: 1.10.5
>Reporter: Yang Hu
>Priority: Minor
>
> Hi Airflow,
> We noticed that sometime, google status url will return us 500 and airflow
> will mark the whole task failed, but is actually runs fine.
>
> I think we need some retry before failing the task, at least at this 500 case
> Sample request:
> > [2019-09-18 21:45:19,543] \{base_task_runner.py:98} INFO - Subtask:
> googleapiclient.errors.HttpError:
> target="_blank">https://datastore.googleapis.com/v1/projects/MASKED/operations/AS_MASKED_RI?alt=json
> returned "Unknown Error.">
> Ref code:
> [https://github.com/apache/airflow/blob/7e4330cce0b4f333b1658cdc315b06505cf9dd76/airflow/gcp/hooks/datastore.py#L269]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5544) GCP datastore operation pull should have some retry to avoid false alarm
[
https://issues.apache.org/jira/browse/AIRFLOW-5544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982972#comment-16982972
]
Yang Hu commented on AIRFLOW-5544:
--
Glad there are fixes there, we are on GCP Cloud Composer so that we have to
wait Google pick up the version which has this update, but thanks for informing
us!
> GCP datastore operation pull should have some retry to avoid false alarm
>
>
> Key: AIRFLOW-5544
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5544
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp
>Affects Versions: 1.10.5
>Reporter: Yang Hu
>Priority: Minor
>
> Hi Airflow,
> We noticed that sometime, google status url will return us 500 and airflow
> will mark the whole task failed, but is actually runs fine.
>
> I think we need some retry before failing the task, at least at this 500 case
> Sample request:
> > [2019-09-18 21:45:19,543] \{base_task_runner.py:98} INFO - Subtask:
> googleapiclient.errors.HttpError:
> target="_blank">https://datastore.googleapis.com/v1/projects/MASKED/operations/AS_MASKED_RI?alt=json
> returned "Unknown Error.">
> Ref code:
> [https://github.com/apache/airflow/blob/7e4330cce0b4f333b1658cdc315b06505cf9dd76/airflow/gcp/hooks/datastore.py#L269]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AIRFLOW-5544) GCP datastore operation pull should have some retry to avoid false alarm
Yang Hu created AIRFLOW-5544:
Summary: GCP datastore operation pull should have some retry to
avoid false alarm
Key: AIRFLOW-5544
URL: https://issues.apache.org/jira/browse/AIRFLOW-5544
Project: Apache Airflow
Issue Type: Bug
Components: gcp
Affects Versions: 1.10.5
Reporter: Yang Hu
Hi Airflow,
We noticed that sometime, google status url will return us 500 and airflow will
mark the whole task failed, but is actually runs fine.
I think we need some retry before failing the task, at least at this 500 case
Sample request:
> [2019-09-18 21:45:19,543] \{base_task_runner.py:98} INFO - Subtask:
googleapiclient.errors.HttpError: https://datastore.googleapis.com/v1/projects/MASKED/operations/AS_MASKED_RI?alt=json
returned "Unknown Error.">
Ref code:
[https://github.com/apache/airflow/blob/7e4330cce0b4f333b1658cdc315b06505cf9dd76/airflow/gcp/hooks/datastore.py#L269]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
