he-gong-sardine commented on code in PR #36545:
URL: https://github.com/apache/airflow/pull/36545#discussion_r1468859918
##########
airflow/providers/google/cloud/transfers/sftp_to_gcs.py:
##########
@@ -168,6 +204,68 @@ def _copy_single_object(
self.log.info("Executing delete of %s", source_path)
sftp_hook.delete_file(source_path)
+ def _stream_single_object(
+ self,
+ sftp_hook: SFTPHook,
+ gcs_hook: GCSHook,
+ source_path: str,
+ destination_object: str
+ ) -> None:
+ """Helper function to stream a single object with robust handling and
logging."""
+ self.log.info(
+ "Starting stream of %s to gs://%s/%s",
+ source_path,
+ self.destination_bucket,
+ destination_object,
+ )
+
+ client = gcs_hook.get_conn()
+ dest_bucket = client.bucket(self.destination_bucket)
+ dest_blob = dest_bucket.blob(destination_object)
+ temp_destination_object = f"{destination_object}.tmp"
+ temp_dest_blob = dest_bucket.blob(temp_destination_object)
+
+ # Check and delete any existing temp file from previous failed attempts
+ if temp_dest_blob.exists():
+ self.log.warning(f"Temporary file {temp_destination_object} found,
deleting for fresh upload.")
Review Comment:
Thank you very much. I didn't know %s and f-string had performance
difference when not called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
