[GitHub] [airflow] david30907d opened a new pull request #19571: :pencil: (providers_google) add a location check

2022-01-08 Thread GitBox 


david30907d opened a new pull request #19571:
URL: https://github.com/apache/airflow/pull/19571


   if you didn't, you'll get this `returned "Not found: Job xxx"` exception
   related: 
   
   Although BigQueryHook says `location` is optional, turns out it's required 
under some scenario
   
[code](https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/hooks/bigquery.py#L99)
   
   when we invoke `get_records()` 
([here](https://github.com/apache/airflow/blob/main/airflow/hooks/dbapi.py#L155)),
  it would end up go to this `run_query()` to get the `job id` 
([here](https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/hooks/bigquery.py#L2202)).
   
   another alternative is raise a `missing location exception` in `run_query()`
   
   thoughts on this?
   ```bash
   Traceback (most recent call last):
 File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py",
 line 1157, in _run_raw_task
   self._prepare_and_execute_task_with_callbacks(context, task)
 File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py",
 line 1331, in _prepare_and_execute_task_with_callbacks
   result = self._execute_task(context, task_copy)
 File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py",
 line 1361, in _execute_task
   result = task_copy.execute(context=context)
 File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", 
line 150, in execute
   return_value = self.execute_callable()
 File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", 
line 161, in execute_callable
   return self.python_callable(*self.op_args, **self.op_kwargs)
 File "/opt/airflow/dags/dags/utils/others/subscription_related.py", line 
112, in wrapper
   return func(*args, **kwargs)
 File 
"/opt/airflow/dags/dags/utils/extractors/platform_data_extractors/shopify_extractor.py",
 line 75, in wrapper
   return func(*args, **kwargs)
 File 
"/opt/airflow/dags/dags/utils/extractors/platform_data_extractors/shopify_extractor.py",
 line 1019, in add_abandoned
   abandoned_checkouts_of_this_page = _parse_this_page(response_json)
 File 
"/opt/airflow/dags/dags/utils/extractors/platform_data_extractors/shopify_extractor.py",
 line 980, in _parse_this_page
   persons_queried_by_checkout_id = db_hook.get_records(
 File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/hooks/dbapi.py", line 
135, in get_records
   return cur.fetchall()
 File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/hooks/bigquery.py",
 line 2886, in fetchall
   one = self.fetchone()
 File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/hooks/bigquery.py",
 line 2811, in fetchone
   return self.next()
 File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/hooks/bigquery.py",
 line 2827, in next
   self.service.jobs()
 File 
"/home/airflow/.local/lib/python3.8/site-packages/googleapiclient/_helpers.py", 
line 134, in positional_wrapper
   return wrapped(*args, **kwargs)
 File 
"/home/airflow/.local/lib/python3.8/site-packages/googleapiclient/http.py", 
line 915, in execute
   raise HttpError(resp, content, uri=self.uri)
   googleapiclient.errors.HttpError: https://bigquery.googleapis.com/bigquery/v2/projects/xx/queries/airflow_x?alt=json
 returned "Not found: Job x:x
   ```
   
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] david30907d opened a new pull request #19571: :pencil: (providers_google) add a location check

2021-11-12 Thread GitBox 


david30907d opened a new pull request #19571:
URL: https://github.com/apache/airflow/pull/19571


   if you didn't, you'll get this `returned "Not found: Job xxx"` exception
   related: 
   
   Although BigQueryHook says `location` is optional, turns out it's required 
under some scenario
   
[code](https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/hooks/bigquery.py#L99)
   
   when we invoke `get_records()` 
([here](https://github.com/apache/airflow/blob/main/airflow/hooks/dbapi.py#L155)),
  it would end up go to this `run_query()` to get the `job id` 
([here](https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/hooks/bigquery.py#L2202)).
   
   another alternative is raise a `missing location exception` in `run_query()`
   
   thoughts on this?
   ```bash
   Traceback (most recent call last):
 File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py",
 line 1157, in _run_raw_task
   self._prepare_and_execute_task_with_callbacks(context, task)
 File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py",
 line 1331, in _prepare_and_execute_task_with_callbacks
   result = self._execute_task(context, task_copy)
 File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py",
 line 1361, in _execute_task
   result = task_copy.execute(context=context)
 File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", 
line 150, in execute
   return_value = self.execute_callable()
 File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", 
line 161, in execute_callable
   return self.python_callable(*self.op_args, **self.op_kwargs)
 File "/opt/airflow/dags/dags/utils/others/subscription_related.py", line 
112, in wrapper
   return func(*args, **kwargs)
 File 
"/opt/airflow/dags/dags/utils/extractors/platform_data_extractors/shopify_extractor.py",
 line 75, in wrapper
   return func(*args, **kwargs)
 File 
"/opt/airflow/dags/dags/utils/extractors/platform_data_extractors/shopify_extractor.py",
 line 1019, in add_abandoned
   abandoned_checkouts_of_this_page = _parse_this_page(response_json)
 File 
"/opt/airflow/dags/dags/utils/extractors/platform_data_extractors/shopify_extractor.py",
 line 980, in _parse_this_page
   persons_queried_by_checkout_id = db_hook.get_records(
 File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/hooks/dbapi.py", line 
135, in get_records
   return cur.fetchall()
 File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/hooks/bigquery.py",
 line 2886, in fetchall
   one = self.fetchone()
 File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/hooks/bigquery.py",
 line 2811, in fetchone
   return self.next()
 File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/hooks/bigquery.py",
 line 2827, in next
   self.service.jobs()
 File 
"/home/airflow/.local/lib/python3.8/site-packages/googleapiclient/_helpers.py", 
line 134, in positional_wrapper
   return wrapped(*args, **kwargs)
 File 
"/home/airflow/.local/lib/python3.8/site-packages/googleapiclient/http.py", 
line 915, in execute
   raise HttpError(resp, content, uri=self.uri)
   googleapiclient.errors.HttpError: https://bigquery.googleapis.com/bigquery/v2/projects/xx/queries/airflow_x?alt=json
 returned "Not found: Job x:x
   ```
   
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org