[jira] [Commented] (AIRFLOW-5544) GCP datastore operation pull should have some retry to avoid false alarm

2019-11-26 Thread Kamil Bregula (Jira) 


[ 
https://issues.apache.org/jira/browse/AIRFLOW-5544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982996#comment-16982996
 ] 

Kamil Bregula commented on AIRFLOW-5544:


There is a plan to make these operators available for Cloud Composer in the 
near future. 
[https://lists.apache.org/thread.html/2c9559184045e772acd21cbdd7435f6bf89c76eb9311311d58d16e5f@%3Cdev.airflow.apache.org%3E]

You will be able to install the latest operators supported by my team.

> GCP datastore operation pull should have some retry to avoid false alarm
> 
>
> Key: AIRFLOW-5544
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5544
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Affects Versions: 1.10.5
>Reporter: Yang Hu
>Priority: Minor
>
> Hi Airflow,
> We noticed that sometime, google status url will return us 500 and airflow 
> will mark the whole task failed, but is actually runs fine.
>  
> I think we need some retry before failing the task, at least at this 500 case
> Sample request:
>  [2019-09-18 21:45:19,543] \{base_task_runner.py:98} INFO - Subtask: 
> googleapiclient.errors.HttpError: HttpError 500 when requesting  href="https://datastore.googleapis.com/v1/projects/MASKED/operations/AS_MASKED_RI?alt=json;
>  
> target="_blank">https://datastore.googleapis.com/v1/projects/MASKED/operations/AS_MASKED_RI?alt=json
>  returned Unknown Error.
> Ref code:
> [https://github.com/apache/airflow/blob/7e4330cce0b4f333b1658cdc315b06505cf9dd76/airflow/gcp/hooks/datastore.py#L269]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5544) GCP datastore operation pull should have some retry to avoid false alarm

2019-11-26 Thread Yang Hu (Jira) 


[ 
https://issues.apache.org/jira/browse/AIRFLOW-5544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982972#comment-16982972
 ] 

Yang Hu commented on AIRFLOW-5544:
--

Glad there are fixes there, we are on GCP Cloud Composer so that we have to 
wait Google pick up the version which has this update, but thanks for informing 
us!

> GCP datastore operation pull should have some retry to avoid false alarm
> 
>
> Key: AIRFLOW-5544
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5544
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Affects Versions: 1.10.5
>Reporter: Yang Hu
>Priority: Minor
>
> Hi Airflow,
> We noticed that sometime, google status url will return us 500 and airflow 
> will mark the whole task failed, but is actually runs fine.
>  
> I think we need some retry before failing the task, at least at this 500 case
> Sample request:
>  [2019-09-18 21:45:19,543] \{base_task_runner.py:98} INFO - Subtask: 
> googleapiclient.errors.HttpError: HttpError 500 when requesting  href="https://datastore.googleapis.com/v1/projects/MASKED/operations/AS_MASKED_RI?alt=json;
>  
> target="_blank">https://datastore.googleapis.com/v1/projects/MASKED/operations/AS_MASKED_RI?alt=json
>  returned Unknown Error.
> Ref code:
> [https://github.com/apache/airflow/blob/7e4330cce0b4f333b1658cdc315b06505cf9dd76/airflow/gcp/hooks/datastore.py#L269]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5544) GCP datastore operation pull should have some retry to avoid false alarm

2019-11-26 Thread Kamil Bregula (Jira) 


[ 
https://issues.apache.org/jira/browse/AIRFLOW-5544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982961#comment-16982961
 ] 

Kamil Bregula commented on AIRFLOW-5544:


Some body from community make change:

[https://github.com/apache/airflow/commit/16e7e6175ae16d73d84c86163686a6c0b9b707ff#diff-d67992de738848a0b4278915ffd9b797]

> GCP datastore operation pull should have some retry to avoid false alarm
> 
>
> Key: AIRFLOW-5544
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5544
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Affects Versions: 1.10.5
>Reporter: Yang Hu
>Priority: Minor
>
> Hi Airflow,
> We noticed that sometime, google status url will return us 500 and airflow 
> will mark the whole task failed, but is actually runs fine.
>  
> I think we need some retry before failing the task, at least at this 500 case
> Sample request:
>  [2019-09-18 21:45:19,543] \{base_task_runner.py:98} INFO - Subtask: 
> googleapiclient.errors.HttpError: HttpError 500 when requesting  href="https://datastore.googleapis.com/v1/projects/MASKED/operations/AS_MASKED_RI?alt=json;
>  
> target="_blank">https://datastore.googleapis.com/v1/projects/MASKED/operations/AS_MASKED_RI?alt=json
>  returned Unknown Error.
> Ref code:
> [https://github.com/apache/airflow/blob/7e4330cce0b4f333b1658cdc315b06505cf9dd76/airflow/gcp/hooks/datastore.py#L269]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5544) GCP datastore operation pull should have some retry to avoid false alarm

2019-11-26 Thread Yang Hu (Jira) 


[ 
https://issues.apache.org/jira/browse/AIRFLOW-5544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982955#comment-16982955
 ] 

Yang Hu commented on AIRFLOW-5544:
--

Hi [~kamil.bregula]

What improvements are there? Do you have a link or version that we can test?

> GCP datastore operation pull should have some retry to avoid false alarm
> 
>
> Key: AIRFLOW-5544
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5544
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Affects Versions: 1.10.5
>Reporter: Yang Hu
>Priority: Minor
>
> Hi Airflow,
> We noticed that sometime, google status url will return us 500 and airflow 
> will mark the whole task failed, but is actually runs fine.
>  
> I think we need some retry before failing the task, at least at this 500 case
> Sample request:
>  [2019-09-18 21:45:19,543] \{base_task_runner.py:98} INFO - Subtask: 
> googleapiclient.errors.HttpError: HttpError 500 when requesting  href="https://datastore.googleapis.com/v1/projects/MASKED/operations/AS_MASKED_RI?alt=json;
>  
> target="_blank">https://datastore.googleapis.com/v1/projects/MASKED/operations/AS_MASKED_RI?alt=json
>  returned Unknown Error.
> Ref code:
> [https://github.com/apache/airflow/blob/7e4330cce0b4f333b1658cdc315b06505cf9dd76/airflow/gcp/hooks/datastore.py#L269]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5544) GCP datastore operation pull should have some retry to avoid false alarm

2019-11-26 Thread Kamil Bregula (Jira) 


[ 
https://issues.apache.org/jira/browse/AIRFLOW-5544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982926#comment-16982926
 ] 

Kamil Bregula commented on AIRFLOW-5544:


[~coolcute] Is it still valid? My team has made improvements in this 
integration.

> GCP datastore operation pull should have some retry to avoid false alarm
> 
>
> Key: AIRFLOW-5544
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5544
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Affects Versions: 1.10.5
>Reporter: Yang Hu
>Priority: Minor
>
> Hi Airflow,
> We noticed that sometime, google status url will return us 500 and airflow 
> will mark the whole task failed, but is actually runs fine.
>  
> I think we need some retry before failing the task, at least at this 500 case
> Sample request:
>  [2019-09-18 21:45:19,543] \{base_task_runner.py:98} INFO - Subtask: 
> googleapiclient.errors.HttpError: HttpError 500 when requesting  href="https://datastore.googleapis.com/v1/projects/MASKED/operations/AS_MASKED_RI?alt=json;
>  
> target="_blank">https://datastore.googleapis.com/v1/projects/MASKED/operations/AS_MASKED_RI?alt=json
>  returned Unknown Error.
> Ref code:
> [https://github.com/apache/airflow/blob/7e4330cce0b4f333b1658cdc315b06505cf9dd76/airflow/gcp/hooks/datastore.py#L269]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)