[
https://issues.apache.org/jira/browse/AIRFLOW-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16787077#comment-16787077
]
ASF subversion and git services commented on AIRFLOW-2985:
----------------------------------------------------------
Commit 1744c0afeaf48244c5d4ee7fcd9b5698b908cd48 in airflow's branch
refs/heads/v1-10-stable from Xiaodong
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=1744c0a ]
[AIRFLOW-2985] Operators for S3 object copying/deleting (#3823)
1. Copying:
Under the hood, it's `boto3.client.copy_object()`.
It can only handle the situation in which the
S3 connection used can access both source and
destination bucket/key.
2. Deleting:
2.1 Under the hood, it's `boto3.client.delete_objects()`.
It supports either deleting one single object or
multiple objects.
2.2 If users try to delete a non-existent object, the
request will still succeed, but there will be an
entry 'Errors' in the response. There may also be
other reasons which may cause similar 'Errors' (
request itself would succeed without explicit
exception). So an argument `silent_on_errors` is added
to let users decide if this sort of 'Errors' should
fail the operator.
The corresponding methods are added into S3Hook, and
these two operators are 'wrappers' of these methods.
> Operators for S3 object copying/deleting
> [boto3.client.copy_object()/delete_object()]
> -------------------------------------------------------------------------------------
>
> Key: AIRFLOW-2985
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2985
> Project: Apache Airflow
> Issue Type: New Feature
> Components: operators
> Reporter: Xiaodong DENG
> Assignee: Xiaodong DENG
> Priority: Minor
> Fix For: 1.10.3
>
>
> Currently we don't have an operator in Airflow to help copy/delete objects
> within S3, while they may be quite common use case when we deal with the data
> in S3.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
