set92 opened a new issue, #28743: URL: https://github.com/apache/airflow/issues/28743
### Apache Airflow version 2.5.0 ### What happened I have a DAG in which I copy files from one bucket to another. Today, a call to s3Hook broke because `s3_hook.copy_object()` can't handle more than 5 Gb, and I have a parquet file of 6Gb. My temporal fix was moving it manually, but I thought a permanent fix would be using `s3_hook.copy` instead of `s3_hook.copy_object`, the problem is that it doesn't exist. In my mind they do the same thing, so if we change in https://github.com/apache/airflow/blob/6cbf9b62e374666693b93f79a36dcb46cc8245c4/airflow/providers/amazon/aws/hooks/s3.py#L814 copy_object to copy wouldn't do the same and also move files bigger than 5 Gb? or there is some other method for this case? The documentation for copy() is in [here](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.copy). ### What you think should happen instead I think it shouldn't matter the size of the file while copying files. In my case, the log it showed was: ``` Traceback (most recent call last): File "/opt/c2/airflow/.venv/lib/python3.9/site-packages/airflow/decorators/base.py", line 188, in execute return_value = super().execute(context) File "/opt/c2/airflow/.venv/lib/python3.9/site-packages/airflow/operators/python.py", line 175, in execute return_value = self.execute_callable() File "/opt/c2/airflow/.venv/lib/python3.9/site-packages/airflow/operators/python.py", line 193, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/opt/c2/airflow/c1/dags/dataengineers/c3/create_dev_env.py", line 32, in copy_input_data _copy_input_data( File "/opt/c2/airflow/c1/dags/dataengineers/c3/actions_create.py", line 155, in _copy_input_data _copy_s3_data(source_bucket, entity["storage"], dest_bucket, folder_prefix) File "/opt/c2/airflow/c1/dags/dataengineers/c3/actions_create.py", line 87, in _copy_s3_data s3_hook.copy_object( File "/opt/c2/airflow/.venv/lib/python3.9/site-packages/airflow/providers/amazon/aws/hooks/s3.py", line 830, in copy_object response = self.get_conn().copy_object( File "/opt/c2/airflow/.venv/lib/python3.9/site-packages/botocore/client.py", line 508, in _api_call return self._make_api_call(operation_name, kwargs) File "/opt/c2/airflow/.venv/lib/python3.9/site-packages/botocore/client.py", line 915, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (InvalidRequest) when calling the CopyObject operation: The specified copy source is larger than the maximum allowable size for a copy source: 5368709120 ``` ### How to reproduce _No response_ ### Operating System ubuntu 20.04 ### Versions of Apache Airflow Providers apache-airflow-providers-amazon==6.0.0 apache-airflow-providers-cncf-kubernetes==4.4.0 apache-airflow-providers-common-sql==1.2.0 apache-airflow-providers-ftp==3.1.0 apache-airflow-providers-google==6.8.0 apache-airflow-providers-http==4.0.0 apache-airflow-providers-imap==3.0.0 apache-airflow-providers-postgres==5.2.2 apache-airflow-providers-sendgrid==3.0.0 apache-airflow-providers-slack==6.0.0 apache-airflow-providers-sqlite==3.2.1 ### Deployment Other 3rd-party Helm chart ### Deployment details _No response_ ### Anything else _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
