set92 opened a new issue, #28743:
URL: https://github.com/apache/airflow/issues/28743

   ### Apache Airflow version
   
   2.5.0
   
   ### What happened
   
   I have a DAG in which I copy files from one bucket to another. Today, a call 
to s3Hook broke because `s3_hook.copy_object()` can't handle more than 5 Gb, 
and I have a parquet file of 6Gb. My temporal fix was moving it manually, but I 
thought a permanent fix would be using `s3_hook.copy` instead of 
`s3_hook.copy_object`, the problem is that it doesn't exist.
   
   In my mind they do the same thing, so if we change in 
https://github.com/apache/airflow/blob/6cbf9b62e374666693b93f79a36dcb46cc8245c4/airflow/providers/amazon/aws/hooks/s3.py#L814
 copy_object to copy wouldn't do the same and also move files bigger than 5 Gb? 
or there is some other method for this case?
   The documentation for copy() is in 
[here](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.copy).
 
   
   
   ### What you think should happen instead
   
   I think it shouldn't matter the size of the file while copying files. In my 
case, the log it showed was: 
   
   ```
   Traceback (most recent call last):
     File 
"/opt/c2/airflow/.venv/lib/python3.9/site-packages/airflow/decorators/base.py", 
line 188, in execute
       return_value = super().execute(context)
     File 
"/opt/c2/airflow/.venv/lib/python3.9/site-packages/airflow/operators/python.py",
 line 175, in execute
       return_value = self.execute_callable()
     File 
"/opt/c2/airflow/.venv/lib/python3.9/site-packages/airflow/operators/python.py",
 line 193, in execute_callable
       return self.python_callable(*self.op_args, **self.op_kwargs)
     File "/opt/c2/airflow/c1/dags/dataengineers/c3/create_dev_env.py", line 
32, in copy_input_data
       _copy_input_data(
     File "/opt/c2/airflow/c1/dags/dataengineers/c3/actions_create.py", line 
155, in _copy_input_data
       _copy_s3_data(source_bucket, entity["storage"], dest_bucket, 
folder_prefix)
     File "/opt/c2/airflow/c1/dags/dataengineers/c3/actions_create.py", line 
87, in _copy_s3_data
       s3_hook.copy_object(
     File 
"/opt/c2/airflow/.venv/lib/python3.9/site-packages/airflow/providers/amazon/aws/hooks/s3.py",
 line 830, in copy_object
       response = self.get_conn().copy_object(
     File 
"/opt/c2/airflow/.venv/lib/python3.9/site-packages/botocore/client.py", line 
508, in _api_call
       return self._make_api_call(operation_name, kwargs)
     File 
"/opt/c2/airflow/.venv/lib/python3.9/site-packages/botocore/client.py", line 
915, in _make_api_call
       raise error_class(parsed_response, operation_name)
   botocore.exceptions.ClientError: An error occurred (InvalidRequest) when 
calling the CopyObject operation: The specified copy source is larger than the 
maximum allowable size for a copy source: 5368709120
   ```
   
   ### How to reproduce
   
   _No response_
   
   ### Operating System
   
   ubuntu 20.04
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-amazon==6.0.0
   apache-airflow-providers-cncf-kubernetes==4.4.0
   apache-airflow-providers-common-sql==1.2.0
   apache-airflow-providers-ftp==3.1.0
   apache-airflow-providers-google==6.8.0
   apache-airflow-providers-http==4.0.0
   apache-airflow-providers-imap==3.0.0
   apache-airflow-providers-postgres==5.2.2
   apache-airflow-providers-sendgrid==3.0.0
   apache-airflow-providers-slack==6.0.0
   apache-airflow-providers-sqlite==3.2.1
   
   ### Deployment
   
   Other 3rd-party Helm chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to