mik-laj commented on a change in pull request #8049: [AIRFLOW-7117] Honor
self.schema in sql_to_gcs as schema to upload
URL: https://github.com/apache/airflow/pull/8049#discussion_r402214684
##########
File path: airflow/providers/google/cloud/operators/sql_to_gcs.py
##########
@@ -251,18 +252,20 @@ def _get_col_type_dict(self):
def _write_local_schema_file(self, cursor):
"""
Takes a cursor, and writes the BigQuery schema for the results to a
- local file system.
+ local file system. Schema for database will be read from cursor if
+ not specified.
:return: A dictionary where key is a filename to be used as an object
name in GCS, and values are file handles to local files that
contains the BigQuery schema fields in .json format.
"""
- schema = [self.field_to_bigquery(field) for field in
cursor.description]
+ schema = self.schema or [self.field_to_bigquery(field) for field in
cursor.description]
self.log.info('Using schema for %s', self.schema_filename)
self.log.debug("Current schema: %s", schema)
tmp_schema_file_handle = NamedTemporaryFile(delete=True)
- tmp_schema_file_handle.write(json.dumps(schema,
sort_keys=True).encode('utf-8'))
+ schema_json = json.dumps(schema, sort_keys=True) if isinstance(schema,
list) else schema
+ tmp_schema_file_handle.write(schema_json.encode('utf-8'))
Review comment:
It looks overly complicated. What do you think about the following code?
```python
if self.shcema:
schema_json = self.schemaa
else:
schema = self.schema or [self.field_to_bigquery(field) for field
in cursor.description]
tmp_schema_file_handle = NamedTemporaryFile(delete=True)
schema_json = json.dumps(schema, sort_keys=True)
self.log.info('Using schema for %s', self.schema_filename)
self.log.debug("Current schema: %s", schema)
tmp_schema_file_handle.write(schema_json.encode('utf-8'))
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
