mik-laj commented on a change in pull request #8049: [AIRFLOW-7117] Honor self.schema in sql_to_gcs as schema to upload URL: https://github.com/apache/airflow/pull/8049#discussion_r402214684
########## File path: airflow/providers/google/cloud/operators/sql_to_gcs.py ########## @@ -251,18 +252,20 @@ def _get_col_type_dict(self): def _write_local_schema_file(self, cursor): """ Takes a cursor, and writes the BigQuery schema for the results to a - local file system. + local file system. Schema for database will be read from cursor if + not specified. :return: A dictionary where key is a filename to be used as an object name in GCS, and values are file handles to local files that contains the BigQuery schema fields in .json format. """ - schema = [self.field_to_bigquery(field) for field in cursor.description] + schema = self.schema or [self.field_to_bigquery(field) for field in cursor.description] self.log.info('Using schema for %s', self.schema_filename) self.log.debug("Current schema: %s", schema) tmp_schema_file_handle = NamedTemporaryFile(delete=True) - tmp_schema_file_handle.write(json.dumps(schema, sort_keys=True).encode('utf-8')) + schema_json = json.dumps(schema, sort_keys=True) if isinstance(schema, list) else schema + tmp_schema_file_handle.write(schema_json.encode('utf-8')) Review comment: It looks overly complicated. What do you think about the following code? ```python if self.shcema: schema_json = self.schemaa else: schema = self.schema or [self.field_to_bigquery(field) for field in cursor.description] tmp_schema_file_handle = NamedTemporaryFile(delete=True) schema_json = json.dumps(schema, sort_keys=True) self.log.info('Using schema for %s', self.schema_filename) self.log.debug("Current schema: %s", schema) tmp_schema_file_handle.write(schema_json.encode('utf-8')) ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services