dinigo commented on issue #8804: URL: https://github.com/apache/airflow/issues/8804#issuecomment-626625086
You are suggesting to implement kind of what we have with [GenericTransfer](https://github.com/apache/airflow/blob/master/airflow/operators/generic_transfer.py). I've thought of this too. We would need to have a common API for getting and sending files. Preferably a `flile-like-object` so transfers are done with a stream and they don't take storage. I would rather have an `XtoYOperator` where I can configure the Hooks for each, source and destination, such as: ``` gcs_to_s3 = GenericFileTransfer( source_hook=GCSHook('gcs-conn-id'), dest_hook=S3Hook('s3-conn-id'), source_file='gs://my-gcs-bucket/my-file.csv', dest_file='my-s3-bucket/my-file.csv' ) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
